[saturna.cluster:19236] progressed_wait: base/plm_base_launch_support.c 459 Daemon was launched on xserve01.cluster - beginning to initialize Daemon [[28467,0],1] checking in as pid 43911 on host xserve01.cluster Daemon [[28467,0],1] not using static ports [saturna.cluster:19236] defining message event: base/plm_base_launch_support.c 423 ! [saturna.cluster:19236] defining message event: grpcomm_bad_module.c 183 [saturna.cluster:19236] progressed_wait: base/plm_base_launch_support.c 712 [saturna.cluster:19236] [[28467,0],0] orte:daemon:cmd:processor called by [[28467,0],0] for tag 1 [saturna.cluster:19236] [[28467,0],0] node[0].name saturna daemon 0 arch ffc90200 [saturna.cluster:19236] [[28467,0],0] node[1].name xserve01 daemon 1 arch ffc90200 [saturna.cluster:19236] [[28467,0],0] orted_cmd: received add_local_procs [saturna.cluster:19236] defining message event: base/odls_base_default_fns.c 1219 [saturna.cluster:19236] [[28467,0],0] orte:daemon:send_relay [saturna.cluster:19236] [[28467,0],0] orte:daemon:send_relay sending relay msg to 1 [xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: received message from [[28467,0],0] [xserve01.cluster:43911] defining message event: orted/orted_comm.c 159 [xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: reissued recv [xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor called by [[28467,0],0] for tag 1 [xserve01.cluster:43911] [[28467,0],1] node[0].name saturna daemon 0 arch ffc90200 [xserve01.cluster:43911] [[28467,0],1] node[1].name xserve01 daemon 1 arch ffc90200 [xserve01.cluster:43911] [[28467,0],1] orted_cmd: received add_local_procs [saturna.cluster:19236] defining message event: base/plm_base_launch_support.c 668 [xserve01.cluster:43911] [[28467,0],1] orte:daemon:send_relay [xserve01.cluster:43911] [[28467,0],1] orte:daemon:send_relay - recipient list is empty! [saturna.cluster:19236] defining message event: iof_hnp_receive.c 227 [saturna.cluster:19236] defining message event: iof_hnp_receive.c 227 MPI init [saturna.cluster:19236] defining message event: iof_hnp_receive.c 227 MPI init [saturna.cluster:19236] defining message event: iof_hnp_receive.c 227 MPI init [saturna.cluster:19236] defining message event: iof_hnp_receive.c 227 MPI init [saturna.cluster:19236] defining message event: iof_hnp_receive.c 227 MPI init [saturna.cluster:19236] defining message event: iof_hnp_receive.c 227 MPI init MPI init [saturna.cluster:19236] defining message event: iof_hnp_receive.c 227 MPI init [xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: received message from [[28467,1],0] [xserve01.cluster:43911] defining message event: orted/orted_comm.c 159 [xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: reissued recv [xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor called by [[28467,1],0] for tag 1 [xserve01.cluster:43911] [[28467,0],1] orted_recv: received sync+nidmap from local proc [[28467,1],0] [xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor: processing commands completed [xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: received message from [[28467,1],1] [xserve01.cluster:43911] defining message event: orted/orted_comm.c 159 [xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: reissued recv [xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor called by [[28467,1],1] for tag 1 [xserve01.cluster:43911] [[28467,0],1] orted_recv: received sync+nidmap from local proc [[28467,1],1] [xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor: processing commands completed [xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: received message from [[28467,1],2] [xserve01.cluster:43911] defining message event: orted/orted_comm.c 159 [xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: reissued recv [xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor called by [[28467,1],2] for tag 1 [xserve01.cluster:43911] [[28467,0],1] orted_recv: received sync+nidmap from local proc [[28467,1],2] [xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor: processing commands completed [xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: received message from [[28467,1],3] [xserve01.cluster:43911] defining message event: orted/orted_comm.c 159 [xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: reissued recv [xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor called by [[28467,1],3] for tag 1 [xserve01.cluster:43911] [[28467,0],1] orted_recv: received sync+nidmap from local proc [[28467,1],3] [xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor: processing commands completed [xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: received message from [[28467,1],4] [xserve01.cluster:43911] defining message event: orted/orted_comm.c 159 [xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: reissued recv [xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor called by [[28467,1],4] for tag 1 [xserve01.cluster:43911] [[28467,0],1] orted_recv: received sync+nidmap from local proc [[28467,1],4] [xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor: processing commands completed [xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: received message from [[28467,1],5] [xserve01.cluster:43911] defining message event: orted/orted_comm.c 159 [xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: reissued recv [xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor called by [[28467,1],5] for tag 1 [xserve01.cluster:43911] [[28467,0],1] orted_recv: received sync+nidmap from local proc [[28467,1],5] [xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor: processing commands completed [xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: received message from [[28467,1],6] [xserve01.cluster:43911] defining message event: orted/orted_comm.c 159 [xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: reissued recv [xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor called by [[28467,1],6] for tag 1 [xserve01.cluster:43911] [[28467,0],1] orted_recv: received sync+nidmap from local proc [[28467,1],6] [xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor: processing commands completed [saturna.cluster:19236] defining message event: base/routed_base_receive.c 153 28467,1],7] [xserve01.cluster:43911] defining message event: orted/orted_comm.c 159 [xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: reissued recv [xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor called by [[28467,1],7] for tag 1 [xserve01.cluster:43911] [[28467,0],1] orted_recv: received sync+nidmap from local proc [[28467,1],7] [xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor: processing commands completed [xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: received message from [[28467,1],2] [xserve01.cluster:43911] defining message event: orted/orted_comm.c 159 [xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: reissued recv [xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor called by [[28467,1],2] for tag 1 [xserve01.cluster:43911] [[28467,0],1] orted_cmd: received collective data cmd [xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor: processing commands completed [xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: received message from [[28467,1],0] [xserve01.cluster:43911] defining message event: orted/orted_comm.c 159 [xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: reissued recv [xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor called by [[28467,1],0] for tag 1 [xserve01.cluster:43911] [[28467,0],1] orted_cmd: received collective data cmd [xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor: processing commands completed [xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: received message from [[28467,1],1] [xserve01.cluster:43911] defining message event: orted/orted_comm.c 159 [xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: reissued recv [xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor called by [[28467,1],1] for tag 1 [xserve01.cluster:43911] [[28467,0],1] orted_cmd: received collective data cmd [xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor: processing commands completed [xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: received message from [[28467,1],3] [xserve01.cluster:43911] defining message event: orted/orted_comm.c 159 [xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: reissued recv [xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor called by [[28467,1],3] for tag 1 [xserve01.cluster:43911] [[28467,0],1] orted_cmd: received collective data cmd [xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor: processing commands completed [xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: received message from [[28467,1],5] [xserve01.cluster:43911] defining message event: orted/orted_comm.c 159 [xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: reissued recv [xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor called by [[28467,1],5] for tag 1 [xserve01.cluster:43911] [[28467,0],1] orted_cmd: received collective data cmd [xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor: processing commands completed [xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: received message from [[28467,1],4] [xserve01.cluster:43911] defining message event: orted/orted_comm.c 159 [xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: reissued recv [xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor called by [[28467,1],4] for tag 1 [xserve01.cluster:43911] [[28467,0],1] orted_cmd: received collective data cmd [xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor: processing commands completed [xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: received message from [[28467,1],6] [xserve01.cluster:43911] defining message event: orted/orted_comm.c 159 [xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: reissued recv [xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor called by [[28467,1],6] for tag 1 [xserve01.cluster:43911] [[28467,0],1] orted_cmd: received collective data cmd [xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor: processing commands completed [saturna.cluster:19236] [[28467,0],0] orted_recv_cmd: received message from [[28467,0],1] [xserve01.cluster:43911] defining message event: orted/orted_comm.c 159 [xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: reissued recv [xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor called by [[28467,1],7] for tag 1 [xserve01.cluster:43911] [[28467,0],1] orted_cmd: received collective data cmd [xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor: proce[saturna.cluster:19236] [[28467,0],0] orted_recv_cmd: reissued recv [saturna.cluster:19236] [[28467,0],0] orte:daemon:cmd:processor called by [[28467,0],1] for tag 1 [saturna.cluster:19236] [[28467,0],0] orted_cmd: received collective data cmd [saturna.cluster:19236] defining message event: grpcomm_bad_module.c 183 [saturna.cluster:19236] [[28467,0],0] orte:daemon:cmd:processor: processing commands completed [saturna.cluster:19236] [[28467,0],0] orte:daemon:cmd:processor called by [[28467,0],0] for tag 1 [saturna.cluster:19236] [[28467,0],0] orted_cmd: received message_local_procs [saturna.cluster:19236] [[28467,0],0] orted:comm:message_local_procs delivering message to job [28467,1] tag 15 [saturna.cluster:19236] [[28467,0],0] orte:daemon:send_relay [saturna.cluster:19236] [[28467,0],0] orte:daemon:send_relay sending relay msg to 1 [xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: received message from [[28467,0],0] [xserve01.cluster:43911] defining message event: orted/orted_comm.c 159 [xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: reissued recv [xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor called by [[28467,0],0] for tag 1 [xserve01.cluster:43911] [[28467,0],1] orted_cmd: received message_local_procs [xserve01.cluster:43911] [[28467,0],1] orted:comm:message_local_procs delivering message to job [28467,1] tag 15 [xserve01.cluster:43911] [[28467,0],1] orte:daemon:send_relay [xserve01.cluster:43911] [[28467,0],1] orte:daemon:send_relay - recipient list is empty! [xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: received message from [[28467,1],5] [xserve01.cluster:43911] defining message event: orted/orted_comm.c 159 [xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: reissued recv [xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor called by [[28467,1],5] for tag 1 [xserve01.cluster:43911] [[28467,0],1] orted_cmd: received collective data cmd [xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor: processing commands completed [xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: received message from [[28467,1],2] [xserve01.cluster:43911] defining message event: orted/orted_comm.c 159 [xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: reissued recv [xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor called by [[28467,1],2] for tag 1 [xserve01.cluster:43911] [[28467,0],1] orted_cmd: received collective data cmd [xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor: processing commands completed [xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: received message from [[28467,1],3] [xserve01.cluster:43911] defining message event: orted/orted_comm.c 159 [xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: reissued recv [xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor called by [[28467,1],3] for tag 1 [xserve01.cluster:43911] [[28467,0],1] orted_cmd: received collective data cmd [xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor: processing commands completed [xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: received message from [[28467,1],4] [xserve01.cluster:43911] defining message event: orted/orted_comm.c 159 [xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: reissued recv [xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor called by [[28467,1],4] for tag 1 [xserve01.cluster:43911] [[28467,0],1] orted_cmd: received collective data cmd [xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor: processing commands completed [xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: received message from [[28467,1],6] [xserve01.cluster:43911] defining message event: orted/orted_comm.c 159 [xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: reissued recv [xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor called by [[28467,1],6] for tag 1 [xserve01.cluster:43911] [[28467,0],1] orted_cmd: received collective data cmd [xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor: processing commands completed [xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: received message from [[28467,1],7] [xserve01.cluster:43911] defining message event: orted/orted_comm.c 159 [xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: reissued recv [xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor called by [[28467,1],7] for tag 1 [xserve01.cluster:43911] [[28467,0],1] orted_cmd: received collective data cmd [xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor: processing commands completed [xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: received message from [[28467,1],0][saturna.cluster:19236] defining message event: orted/orted_comm.c 159 9 [xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: reissued recv [xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor called by [[28467,1],0] for tag 1 [xserve01.cluster:43911] [[28467,0],1] orted_cmd: received collective data cmd [xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor: processing commands completed [xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: received message from [[28467,1],1] [xserve01.cluster:43911] defining message event: orted/orted_comm.c 159 [xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: reissued recv [xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor called by [[28467,1],1] for tag 1 [xserve01.cluster:43911] [[28467,0],1] orted_cmd: received collective data cmd [xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor: proce[saturna.cluster:19236] [[28467,0],0] orted_recv_cmd: reissued recv [saturna.cluster:19236] [[28467,0],0] orte:daemon:cmd:processor called by [[28467,0],1] for tag 1 [saturna.cluster:19236] [[28467,0],0] orted_cmd: received collective data cmd [saturna.cluster:19236] defining message event: grpcomm_bad_module.c 183 [saturna.cluster:19236] [[28467,0],0] orte:daemon:cmd:processor: processing commands completed [saturna.cluster:19236] [[28467,0],0] orte:daemon:cmd:processor called by [[28467,0],0] for tag 1 [saturna.cluster:19236] [[28467,0],0] orted_cmd: received message_local_procs [saturna.cluster:19236] [[28467,0],0] orted:comm:message_local_procs delivering message to job [28467,1] tag 17 [saturna.cluster:19236] [[28467,0],0] orte:daemon:send_relay [saturna.cluster:19236] [[28467,0],0] orte:daemon:send_relay sending relay msg to 1 [xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: received message from [[28467,0],0] [xserve01.cluster:43911] defining message event: orted/orted_comm.c 159 [xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: reissued recv [xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor called by [[28467,0],0] for tag 1 [xserve01.cluster:43911] [[28467,0],1] orted_cmd: received message_local_procs [xserve01.cluster:43911] [[28467,0],1] orted:comm:message_local_procs delivering message to job [28467,1] tag 17 [xserve01.cluster:43911] [[28467,0],1] orte:daemon:send_relay [xserve01.cluster:43911] [[28467,0],1] orte:daemon:send_relay - recipient list is empty! [saturna.cluster:19236] defining message event: iof_hnp_receive.c 227 Done MPI init [saturna.cluster:19236] defining message event: iof_hnp_receive.c 227 Done MPI init [saturna.cluster:19236] defining message event: iof_hnp_receive.c 227 [saturna.cluster:19236] defining message event: iof_hnp_receive.c 227 Done MPI init [saturna.cluster:19236] defining message event: iof_hnp_receive.c 227 Done MPI init checking connection between rank 0 on xserve01.cluster and rank 1 [saturna.cluster:19236] defining message event: iof_hnp_receive.c 227 Done MPI init [saturna.cluster:19236] defining message event: iof_hnp_receive.c 227 checking connection between rank 0 on xserve01.cluster and rank 2 [saturna.cluster:19236] defining message event: iof_hnp_receive.c 227 checking connection between rank 1 on xserve01.cluster and rank 2 [saturna.cluster:19236] defining message event: iof_hnp_receive.c 227 checking connection between rank 0 on xserve01.cluster and rank 3 [saturna.cluster:19236] defining message event: iof_hnp_receive.c 227 checking connection between rank 1 on xserve01.cluster and rank 3 [saturna.cluster:19236] defining message event: iof_hnp_receive.c 227 checking connection between rank 2 on xserve01.cluster and rank 3 [saturna.cluster:19236] defining message event: iof_hnp_receive.c 227 Done MPI init [saturna.cluster:19236] defining message event: iof_hnp_receive.c 227 Done MPI init [saturna.cluster:19236] defining message event: iof_hnp_receive.c 227 Done MPI init [saturna.cluster:19236] defining message event: iof_hnp_receive.c 227 checking connection between rank 0 on xserve01.cluster and rank 4 [saturna.cluster:19236] defining message event: iof_hnp_receive.c 227 checking connection between rank 1 on xserve01.cluster and rank 4 [saturna.cluster:19236] defining message event: iof_hnp_receive.c 227 checking connection between rank 2 on xserve01.cluster and rank 4 [saturna.cluster:19236] defining message event: iof_hnp_receive.c 227 checking connection between rank 3 on xserve01.cluster and rank 4 [saturna.cluster:19236] defining message event: iof_hnp_receive.c 227 checking connection between rank 0 on xserve01.cluster and rank 5 checking connection between rank 0 on xserve01.cluster and rank 6 : orted/orted_comm.c 159 [xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: reissued recv [xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor called by [[28467,1],0] for tag 1 [xserve01.cluster:43911] [[28467,0],1] orted_cmd: received collective data cmd [xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor: processing commands completed [xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: received message from [[28467,1],1] [xserve01.cluster:43911] defining message event[saturna.cluster:19236] defining message event: iof_hnp_receive.c 227 recv_cmd: reissued recv [xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor call[saturna.cluster:19236] defining message event: iof_hnp_receive.c 227 rted_cmd: received collective data cmd [xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor: processing commands completed [xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: received messa[saturna.cluster:19236] defining message event: iof_hnp_receive.c 227 orted/orted_comm.c 159 [xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: reissued recv [xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor called by [[28467,1],2] for tag 1 [xserve01.cluster:43911] [[28467,0],1] orted_cmd: received collective data cmd [xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor: processing commands completed [xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: received message from [[28467,1],3] [xserve01.cluster:43911] defining message event: orted/orted_comm.c 159 [xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: reissued recv [xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor called by [[28467,1],3] for tag 1 [xserve01.cluster:43911] [[28467,0],1] orted_cmd: received collective data cmd [xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor: processing commands completed [xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: received message from [[28467,1],4] [xserve01.cluster:43911] defining message event: orted/orted_comm.c 159 [xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: reissued recv [xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor called by [[28467,1],4] for tag 1 [xserve01.cluster:43911] [[28467,0],1] orted_cmd: received collective data cmd [xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor: processing commands completed [xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: received message from [[28467,1],5] [xserve01.cluster:43911] defining message event: orted/orted_comm.c 159 [xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: reissued recv [xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor called by [[28467,1],5] for tag 1 [xserve01.cluster:43911] [[28467,0],1] orted_cmd: received collective data cmd [xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor: processing commands completed [xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: received message from [[28467,1],6] [xserve01.cluster:43911] defining message event: orted/orted_comm.c 159 checking connection between rank 2 on xserve01.cluster and rank 5 checking connection between rank 2 on xserve01.cluster and rank 6 ed by [[28467,1],6] for tag 1 [xserve01.cluster:43911] [[28467,0],1] orted_cmd: received collective data cmd [xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor: processing commands completed [xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: received message from [[28467,1],7] [xserve01.cluster:43911] defining message event: orted/orted_comm.c 159 [xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: reissued recv [xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor called by [[28467,1],7] for tag 1 [xserve01.cluster:43911] [[28467,0],1] orted_cmd: received collective data cmd [xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor: processing commands completed [saturna.cluster:19236] defining message event: iof_hnp_receive.c 227 checking connection between rank 3 on xserve01.cluster and rank 5 checking connection between rank 3 on xserve01.cluster and rank 6 [saturna.cluster:19236] defining message event: iof_hnp_receive.c 227 checking connection between rank 4 on xserve01.cluster and rank 5 checking connection between rank 4 on xserve01.cluster and rank 6 [saturna.cluster:19236] defining message event: iof_hnp_receive.c 227 checking connection between rank 0 on xserve01.cluster and rank 7 [saturna.cluster:19236] defining message event: iof_hnp_receive.c 227 checking connection between rank 1 on xserve01.cluster and rank 7 [saturna.cluster:19236] defining message event: iof_hnp_receive.c 227 checking connection between rank 2 on xserve01.cluster and rank 7 [saturna.cluster:19236] defining message event: iof_hnp_receive.c 227 checking connection between rank 3 on xserve01.cluster and rank 7 [saturna.cluster:19236] defining message event: iof_hnp_receive.c 227 checking connection between rank 5 on xserve01.cluster and rank 6 checking connection between rank 5 on xserve01.cluster and rank 7 [saturna.cluster:19236] defining message event: iof_hnp_receive.c 227 checking connection between rank 4 on xserve01.cluster and rank 7 [saturna.cluster:19236] defining message event: iof_hnp_receive.c 227 checking connection between rank 6 on xserve01.cluster and rank 7 [saturna.cluster:19236] [[28467,0],0] orted_recv_cmd: received message from [[28467,0],1] [saturna.cluster:19236] defining message event: orted/orted_comm.c 159 [saturna.cluster:19236] [[28467,0],0] orted_recv_cmd: reissued recv Connectivity test on 8 processes PASSED. [saturna.cluster:19236] [[28467,0],0] orte:daemon:cmd:processor called by [[28467,0],1] for tag 1 [saturna.cluster:19236] [[28467,0],0] orted_cmd: received collective data cmd [saturna.cluster:19236] defining message event: grpcomm_bad_module.c 183 [saturna.cluster:19236] [[28467,0],0] orte:daemon:cmd:processor: processing commands completed [saturna.cluster:19236] [[28467,0],0] orte:daemon:cmd:processor called by [[28467,0],0] for tag 1 [saturna.cluster:19236] [[28467,0],0] orted_cmd: received message_local_procs [saturna.cluster:19236] [[28467,0],0] orted:comm:message_local_procs delivering message to job [28467,1] tag 17 [saturna.cluster:19236] [[28467,0],0] orte:daemon:send_relay [saturna.cluster:19236] [[28467,0],0] orte:daemon:send_relay sending relay msg to 1 [xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: received message from [[28467,0],0] [xserve01.cluster:43911] defining message event: orted/orted_comm.c 159 [xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: reissued recv [xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor called by [[28467,0],0] for tag 1 [xserve01.cluster:43911] [[28467,0],1] orted_cmd: received message_local_procs [xserve01.cluster:43911] [[28467,0],1] orted:comm:message_local_procs delivering message to job [28467,1] tag 17 [xserve01.cluster:43911] [[28467,0],1] orte:daemon:send_relay [xserve01.cluster:43911] [[28467,0],1] orte:daemon:send_relay - recipient list is empty! [xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: received message from [[28467,1],0] [xserve01.cluster:43911] defining message event: orted/orted_comm.c 159 [xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: reissued recv [xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor called by [[28467,1],0] for tag 1 [xserve01.cluster:43911] [[28467,0],1] orted_recv: received sync from local proc [[28467,1],0] [xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor: processing commands completed [xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: received message from [[28467,1],2] [xserve01.cluster:43911] defining message event: orted/orted_comm.c 159 [xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: reissued recv [xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor called by [[28467,1],2] for tag 1 [xserve01.cluster:43911] [[28467,0],1] orted_recv: received sync from local proc [[28467,1],2] [xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor: processing commands completed [xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: received message from [[28467,1],4] [xserve01.cluster:43911] defining message event: orted/orted_comm.c 159 [xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: reissued recv [xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor called by [[28467,1],4] for tag 1 [xserve01.cluster:43911] [[28467,0],1] orted_recv: received sync from local proc [[28467,1],4] [xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor: processing commands completed [xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: received message from [[28467,1],5] [xserve01.cluster:43911] defining message event: orted/orted_comm.c 159 [xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: reissued recv [xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor called by [[28467,1],5] for tag 1 [xserve01.cluster:43911] [[28467,0],1] orted_recv: received sync from local proc [[28467,1],5] [xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor: processing commands completed [xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: received message from [[28467,1],6] [xserve01.cluster:43911] defining message event: orted/orted_comm.c 159 [xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: reissued recv [xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor called by [[28467,1],6] for tag 1 [xserve01.cluster:43911] [[28467,0],1] orted_recv: received sync from local proc [[28467,1],6] [xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor: processing commands completed [xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: received message from [[28467,1],7] [xserve01.cluster:43911] defining message event: orted/orted_comm.c 159 [xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: reissued recv [xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor called by [[28467,1],7] for tag 1 [xserve01.cluster:43911] [[28467,0],1] orted_recv: received sync from local proc [[28467,1],7] [xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor: processing commands completed [xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: received message from [[28467,1],3] [xserve01.cluster:43911] defining message event: orted/orted_comm.c 159 [xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: reissued recv [xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor called by [[28467,1],3] for tag 1 [xserve01.cluster:43911] [[28467,0],1] orted_recv: received sync from local proc [[28467,1],3] [xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor: processing commands completed [xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: received message from [[28467,1],1] [xserve01.cluster:43911] defining message event: orted/orted_comm.c 159 [xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: reissued recv [xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor called by [[28467,1],1] for tag 1 [xserve01.cluster:43911] [[28467,0],1] orted_recv: received sync from local proc [[28467,1],1] [xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor: processing commands completed [xserve01.cluster:43911] defining message event: base/odls_base_default_fns.c 2055 [xserve01.cluster:43911] defining message event: iof_orted_read.c 211 [xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor called by [[28467,0],1] for tag 1 [xserve01.cluster:43911] [[28467,0],1] orted_cmd: received waitpid_fired cmd [xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor: processing commands completed [xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor called by [[28467,0],1] for tag 1 [xserve01.cluster:43911] [[28467,0],1] orted_cmd: received iof_complete cmd [xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor: processing commands completed [xserve01.cluster:43911] defining message event: base/odls_base_default_fns.c 2055 [xserve01.cluster:43911] defining message event: iof_orted_read.c 211 [xserve01.cluster:43911] defining message event: base/odls_base_default_fns.c 2055 [xserve01.cluster:43911] defining message event: iof_orted_read.c 211 [xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor called by [[28467,0],1] for tag 1 [xserve01.cluster:43911] [[28467,0],1] orted_cmd: received waitpid_fired cmd [xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor: processing commands completed [xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor called by [[28467,0],1] for tag 1 [xserve01.cluster:43911] [[28467,0],1] orted_cmd: received iof_complete cmd [xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor: processing commands completed [xserve01.cluster:43911] defining message event: base/odls_base_default_fns.c 2055 [xserve01.cluster:43911] defining message event: base/odls_base_default_fns.c 2055 [xserve01.cluster:43911] defining message event: base/odls_base_default_fns.c 2055 [xserve01.cluster:43911] defining message event: base/odls_base_default_fns.c 2055 [xserve01.cluster:43911] defining message event: base/odls_base_default_fns.c 2055 [xserve01.cluster:43911] defining message event: iof_orted_read.c 211 [xserve01.cluster:43911] defining message event: iof_orted_read.c 211 [xserve01.cluster:43911] defining message event: iof_orted_read.c 211 [xserve01.cluster:43911] defining message event: iof_orted_read.c 211 [xserve01.cluster:43911] defining message event: iof_orted_read.c 211 [xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor called by [[28467,0],1] for tag 1 [xserve01.cluster:43911] [[28467,0],1] orted_cmd: received waitpid_fired cmd [xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor: processing commands completed [xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor called by [[28467,0],1] for tag 1 [xserve01.cluster:43911] [[28467,0],1] orted_cmd: received iof_complete cmd [xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor: processing commands completed [xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor called by [[28467,0],1] for tag 1 [xserve01.cluster:43911] [[28467,0],1] orted_cmd: received waitpid_fired cmd [xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor: processing commands completed [xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor called by [[28467,0],1] for tag 1 [xserve01.cluster:43911] [[28467,0],1] orted_cmd: received waitpid_fired cmd [xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor: processing commands completed [xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor called by [[28467,0],1] for tag 1 [xserve01.cluster:43911] [[28467,0],1] orted_cmd: received waitpid_fired cmd [xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor: processing commands completed [xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor called by [[28467,0],1] for tag 1 [xserve01.cluster:43911] [[28467,0],1] orted_cmd: received waitpid_fired cmd [xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor: processing commands completed [xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor called by [[28467,0],1] for tag 1 [xserve01.cluster:43911] [[28467,0],1] orted_cmd: received waitpid_fired cmd [xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor: processing commands completed [xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor called by [[28467,0],1] for tag 1 [xserve01.cluster:43911] [[28467,0],1] orted_cmd: received iof_complete cmd [xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor: processing commands completed [xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor called by [[28467,0],1] for tag 1 [xserve01.cluster:43911] [[28467,0],1] orted_cmd: received iof_complete cmd [xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor: processing commands completed [xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor called by [[28467,0],1] for tag 1 [xserve01.cluster:43911] [[28467,0],1] orted_cmd: received iof_complete cmd [xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor: processing commands completed [xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor called by [[28467,0],1] for tag 1 [xserve01.cluster:43911] [[28467,0],1] orted_cmd: received iof_complete cmd [xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor: processing commands completed [xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor called by [[28467,0],1] for tag 1 [xserve01.cluster:43911] [[28467,0],1] orted_cmd: received iof_complete cmd [saturna.cluster:19236] defining message event: base/plm_base_receive.c 327 commands completed [saturna.cluster:19236] [[28467,0],0] calling job_complete trigger [saturna.cluster:19236] defining message event: grpcomm_bad_module.c 183 [saturna.cluster:19236] [[28467,0],0] orte:daemon:cmd:processor called by [[28467,0],0] for tag 1 [saturna.cluster:19236] [[28467,0],0] orted_cmd: received exit [saturna.cluster:19236] [[28467,0],0] orte:daemon:send_relay [saturna.cluster:19236] [[28467,0],0] orte:daemon:send_relay sending relay msg to 1 [xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: received message from[saturna.cluster:19236] [[28467,0],0] calling orted_exit trigger rted/orted_comm.c 159 [xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: reissued recv [xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor called by [[28467,0],0] for tag 1 [xserve01.cluster:43911] [[28467,0],1] orted_cmd: received exit [xserve01.cluster:43911] [[28467,0],1] orte:daemon:send_relay [xserve01.cluster:43911] [[28467,0],1] orte:daemon:send_relay - recipient list is empty! [xserve01.cluster:43911] [[28467,0],1] calling orted_shutdown trigger [xserve01.cluster:43911] [[28467,0],1] orted: finalizing