[uinta-0061:11195] *** An error occurred in MPI_Comm_rank
[uinta-0061:11195] *** on communicator MPI_COMM_WORLD
[uinta-0061:11195] *** MPI_ERR_COMM: invalid communicator
[uinta-0061:11195] *** MPI_ERRORS_ARE_FATAL (goodbye)
[uinta-0061:11196] *** An error occurred in MPI_Comm_rank
[uinta-0056:27039] *** An error occurred in MPI_Comm_rank
[uinta-0056:27039] *** on communicator MPI_COMM_WORLD
[uinta-0056:27039] *** MPI_ERR_COMM: invalid communicator
[uinta-0056:27039] *** MPI_ERRORS_ARE_FATAL (goodbye)
[uinta-0054:04645] *** An error occurred in MPI_Comm_rank
[uinta-0054:04645] *** on communicator MPI_COMM_WORLD
[uinta-0054:04645] *** MPI_ERR_COMM: invalid communicator
[uinta-0054:04645] *** MPI_ERRORS_ARE_FATAL (goodbye)
[uinta-0061:11194] *** An error occurred in MPI_Comm_rank
[uinta-0061:11194] *** on communicator MPI_COMM_WORLD
[uinta-0061:11194] *** MPI_ERR_COMM: invalid communicator
[uinta-0061:11194] *** MPI_ERRORS_ARE_FATAL (goodbye)
[uinta-0052:26683] *** An error occurred in MPI_Comm_rank
[uinta-0052:26683] *** on communicator MPI_COMM_WORLD
[uinta-0052:26683] *** MPI_ERR_COMM: invalid communicator
[uinta-0052:26683] *** MPI_ERRORS_ARE_FATAL (goodbye)
[uinta-0061:11196] *** on communicator MPI_COMM_WORLD
[uinta-0061:11196] *** MPI_ERR_COMM: invalid communicator
[uinta-0054:04647] *** An error occurred in MPI_Comm_rank
[uinta-0052:26686] *** An error occurred in MPI_Comm_rank
[uinta-0061:11196] *** MPI_ERRORS_ARE_FATAL (goodbye)
[uinta-0056:27038] *** An error occurred in MPI_Comm_rank
[uinta-0056:27038] *** on communicator MPI_COMM_WORLD
[uinta-0056:27038] *** MPI_ERR_COMM: invalid communicator
[uinta-0056:27038] *** MPI_ERRORS_ARE_FATAL (goodbye)
[uinta-0054:04646] *** An error occurred in MPI_Comm_rank
[uinta-0052:26686] *** on communicator MPI_COMM_WORLD
[uinta-0054:04646] *** on communicator MPI_COMM_WORLD
[uinta-0052:26686] *** MPI_ERR_COMM: invalid communicator
[uinta-0061:11197] *** An error occurred in MPI_Comm_rank
[uinta-0056:27036] *** An error occurred in MPI_Comm_rank
[uinta-0054:04647] *** on communicator MPI_COMM_WORLD
[uinta-0061:11197] *** on communicator MPI_COMM_WORLD
[uinta-0056:27036] *** on communicator MPI_COMM_WORLD
[uinta-0054:04646] *** MPI_ERR_COMM: invalid communicator
[uinta-0061:11197] *** MPI_ERR_COMM: invalid communicator
[uinta-0056:27036] *** MPI_ERR_COMM: invalid communicator
[uinta-0054:04647] *** MPI_ERR_COMM: invalid communicator
[uinta-0061:11197] *** MPI_ERRORS_ARE_FATAL (goodbye)
[uinta-0056:27036] *** MPI_ERRORS_ARE_FATAL (goodbye)
[uinta-0054:04646] *** MPI_ERRORS_ARE_FATAL (goodbye)
[uinta-0052:26685] *** An error occurred in MPI_Comm_rank
[uinta-0054:04647] *** MPI_ERRORS_ARE_FATAL (goodbye)
[uinta-0052:26686] *** MPI_ERRORS_ARE_FATAL (goodbye)
[uinta-0052:26684] *** An error occurred in MPI_Comm_rank
[uinta-0052:26685] *** on communicator MPI_COMM_WORLD
[uinta-0056:27037] *** An error occurred in MPI_Comm_rank
[uinta-0056:27037] *** on communicator MPI_COMM_WORLD
[uinta-0052:26684] *** on communicator MPI_COMM_WORLD
[uinta-0056:27037] *** MPI_ERR_COMM: invalid communicator
[uinta-0052:26685] *** MPI_ERR_COMM: invalid communicator
[uinta-0056:27037] *** MPI_ERRORS_ARE_FATAL (goodbye)
[uinta-0052:26684] *** MPI_ERR_COMM: invalid communicator
[uinta-0054:04644] *** An error occurred in MPI_Comm_rank
[uinta-0052:26685] *** MPI_ERRORS_ARE_FATAL (goodbye)
[uinta-0054:04644] *** on communicator MPI_COMM_WORLD
[uinta-0054:04644] *** MPI_ERR_COMM: invalid communicator
[uinta-0054:04644] *** MPI_ERRORS_ARE_FATAL (goodbye)
[uinta-0052:26684] *** MPI_ERRORS_ARE_FATAL (goodbye)
[uinta-0061:11191] [0,0,0] ORTE_ERROR_LOG: Timeout in file ../../../../orte/mca/pls/base/pls_base_orted_cmds.c at line 275
[uinta-0061:11191] [0,0,0] ORTE_ERROR_LOG: Timeout in file ../../../../../orte/mca/pls/tm/pls_tm_module.c at line 572
[uinta-0061:11191] [0,0,0] ORTE_ERROR_LOG: Timeout in file ../../../../../orte/mca/errmgr/hnp/errmgr_hnp.c at line 90
[uinta-0061:11191] [0,0,0] ORTE_ERROR_LOG: Timeout in file ../../../../orte/mca/pls/base/pls_base_orted_cmds.c at line 188
[uinta-0061:11191] [0,0,0] ORTE_ERROR_LOG: Timeout in file ../../../../../orte/mca/pls/tm/pls_tm_module.c at line 603
--------------------------------------------------------------------------
mpiexec was unable to cleanly terminate the daemons for this job. Returned value Timeout instead of ORTE_SUCCESS.
--------------------------------------------------------------------------
[uinta-0052:26679] OOB: Connection to HNP lost