--------------------------------------------------------------------------
[0,1,11]: Myrinet/GM on host wasatch-55 was unable to find any NICs.
Another transport will be used instead, although this may result in 
lower performance.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
[0,1,12]: Myrinet/GM on host wasatch-55 was unable to find any NICs.
Another transport will be used instead, although this may result in 
lower performance.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
[0,1,3]: Myrinet/GM on host wasatch-56 was unable to find any NICs.
Another transport will be used instead, although this may result in 
lower performance.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
[0,1,4]: Myrinet/GM on host wasatch-56 was unable to find any NICs.
Another transport will be used instead, although this may result in 
lower performance.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
[0,1,2]: Myrinet/GM on host wasatch-56 was unable to find any NICs.
Another transport will be used instead, although this may result in 
lower performance.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
[0,1,7]: Myrinet/GM on host wasatch-56 was unable to find any NICs.
Another transport will be used instead, although this may result in 
lower performance.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
[0,1,1]: Myrinet/GM on host wasatch-56 was unable to find any NICs.
Another transport will be used instead, although this may result in 
lower performance.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
[0,1,10]: Myrinet/GM on host wasatch-55 was unable to find any NICs.
Another transport will be used instead, although this may result in 
lower performance.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
[0,1,6]: Myrinet/GM on host wasatch-56 was unable to find any NICs.
Another transport will be used instead, although this may result in 
lower performance.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
[0,1,14]: Myrinet/GM on host wasatch-55 was unable to find any NICs.
Another transport will be used instead, although this may result in 
lower performance.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
[0,1,8]: Myrinet/GM on host wasatch-55 was unable to find any NICs.
Another transport will be used instead, although this may result in 
lower performance.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
[0,1,9]: Myrinet/GM on host wasatch-55 was unable to find any NICs.
Another transport will be used instead, although this may result in 
lower performance.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
[0,1,5]: Myrinet/GM on host wasatch-56 was unable to find any NICs.
Another transport will be used instead, although this may result in 
lower performance.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
[0,1,13]: Myrinet/GM on host wasatch-55 was unable to find any NICs.
Another transport will be used instead, although this may result in 
lower performance.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
[0,1,0]: Myrinet/GM on host wasatch-56 was unable to find any NICs.
Another transport will be used instead, although this may result in 
lower performance.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
[0,1,15]: Myrinet/GM on host wasatch-55 was unable to find any NICs.
Another transport will be used instead, although this may result in 
lower performance.
--------------------------------------------------------------------------
[wasatch-56:18793] *** An error occurred in MPI_Comm_rank
[wasatch-56:18793] *** on communicator MPI_COMM_WORLD
[wasatch-56:18793] *** MPI_ERR_COMM: invalid communicator
[wasatch-56:18793] *** MPI_ERRORS_ARE_FATAL (goodbye)
[wasatch-56:18794] *** An error occurred in MPI_Comm_rank
[wasatch-56:18794] *** on communicator MPI_COMM_WORLD
[wasatch-56:18794] *** MPI_ERR_COMM: invalid communicator
[wasatch-56:18794] *** MPI_ERRORS_ARE_FATAL (goodbye)
[wasatch-56:18795] *** An error occurred in MPI_Comm_rank
[wasatch-56:18795] *** on communicator MPI_COMM_WORLD
[wasatch-56:18795] *** MPI_ERR_COMM: invalid communicator
[wasatch-56:18795] *** MPI_ERRORS_ARE_FATAL (goodbye)
[wasatch-55:14670] *** An error occurred in MPI_Comm_rank
[wasatch-55:14670] *** on communicator MPI_COMM_WORLD
[wasatch-55:14670] *** MPI_ERR_COMM: invalid communicator
[wasatch-55:14670] *** MPI_ERRORS_ARE_FATAL (goodbye)
[wasatch-55:14671] *** An error occurred in MPI_Comm_rank
[wasatch-55:14671] *** on communicator MPI_COMM_WORLD
[wasatch-55:14671] *** MPI_ERR_COMM: invalid communicator
[wasatch-55:14671] *** MPI_ERRORS_ARE_FATAL (goodbye)
[wasatch-55:14677] *** An error occurred in MPI_Comm_rank
[wasatch-55:14677] *** on communicator MPI_COMM_WORLD
[wasatch-55:14677] *** MPI_ERR_COMM: invalid communicator
[wasatch-55:14677] *** MPI_ERRORS_ARE_FATAL (goodbye)
[wasatch-56:18798] *** An error occurred in MPI_Comm_rank
[wasatch-55:14676] *** An error occurred in MPI_Comm_rank
[wasatch-55:14676] *** on communicator MPI_COMM_WORLD
[wasatch-55:14676] *** MPI_ERR_COMM: invalid communicator
[wasatch-55:14676] *** MPI_ERRORS_ARE_FATAL (goodbye)
[wasatch-56:18798] *** on communicator MPI_COMM_WORLD
[wasatch-56:18798] *** MPI_ERR_COMM: invalid communicator
[wasatch-56:18798] *** MPI_ERRORS_ARE_FATAL (goodbye)
[wasatch-55:14673] *** An error occurred in MPI_Comm_rank
[wasatch-55:14673] *** on communicator MPI_COMM_WORLD
[wasatch-55:14673] *** MPI_ERR_COMM: invalid communicator
[wasatch-55:14673] *** MPI_ERRORS_ARE_FATAL (goodbye)
[wasatch-55:14674] *** An error occurred in MPI_Comm_rank
[wasatch-55:14674] *** on communicator MPI_COMM_WORLD
[wasatch-55:14674] *** MPI_ERR_COMM: invalid communicator
[wasatch-55:14674] *** MPI_ERRORS_ARE_FATAL (goodbye)
[wasatch-56:18799] *** An error occurred in MPI_Comm_rank
[wasatch-55:14675] *** An error occurred in MPI_Comm_rank
[wasatch-55:14675] *** on communicator MPI_COMM_WORLD
[wasatch-55:14675] *** MPI_ERR_COMM: invalid communicator
[wasatch-55:14675] *** MPI_ERRORS_ARE_FATAL (goodbye)
[wasatch-56:18799] *** on communicator MPI_COMM_WORLD
[wasatch-56:18799] *** MPI_ERR_COMM: invalid communicator
[wasatch-56:18799] *** MPI_ERRORS_ARE_FATAL (goodbye)
[wasatch-55:14672] *** An error occurred in MPI_Comm_rank
[wasatch-55:14672] *** on communicator MPI_COMM_WORLD
[wasatch-55:14672] *** MPI_ERR_COMM: invalid communicator
[wasatch-55:14672] *** MPI_ERRORS_ARE_FATAL (goodbye)
[wasatch-56:18800] *** An error occurred in MPI_Comm_rank
[wasatch-56:18800] *** on communicator MPI_COMM_WORLD
[wasatch-56:18800] *** MPI_ERR_COMM: invalid communicator
[wasatch-56:18800] *** MPI_ERRORS_ARE_FATAL (goodbye)
[wasatch-56:18797] *** An error occurred in MPI_Comm_rank
[wasatch-56:18797] *** on communicator MPI_COMM_WORLD
[wasatch-56:18797] *** MPI_ERR_COMM: invalid communicator
[wasatch-56:18797] *** MPI_ERRORS_ARE_FATAL (goodbye)
[wasatch-56:18796] *** An error occurred in MPI_Comm_rank
[wasatch-56:18796] *** on communicator MPI_COMM_WORLD
[wasatch-56:18796] *** MPI_ERR_COMM: invalid communicator
[wasatch-56:18796] *** MPI_ERRORS_ARE_FATAL (goodbye)
[wasatch-56:18790] [0,0,0] ORTE_ERROR_LOG: Timeout in file ../../../../orte/mca/pls/base/pls_base_orted_cmds.c at line 275
[wasatch-56:18790] [0,0,0] ORTE_ERROR_LOG: Timeout in file ../../../../../orte/mca/pls/tm/pls_tm_module.c at line 572
[wasatch-56:18790] [0,0,0] ORTE_ERROR_LOG: Timeout in file ../../../../../orte/mca/errmgr/hnp/errmgr_hnp.c at line 90
mpiexec noticed that job rank 10 with PID 14672 on node wasatch-55 exited on signal 15 (Terminated). 
[wasatch-56:18790] [0,0,0] ORTE_ERROR_LOG: Timeout in file ../../../../orte/mca/pls/base/pls_base_orted_cmds.c at line 188
[wasatch-56:18790] [0,0,0] ORTE_ERROR_LOG: Timeout in file ../../../../../orte/mca/pls/tm/pls_tm_module.c at line 603
--------------------------------------------------------------------------
mpiexec was unable to cleanly terminate the daemons for this job. Returned value Timeout instead of ORTE_SUCCESS.
--------------------------------------------------------------------------
[wasatch-56:18792] OOB: Connection to HNP lost