[saturna.cluster:19337] progressed_wait: base/plm_base_launch_support.c 459 Daemon was launched on xserve02.local - beginning to initialize Daemon was launched on xserve01.cluster - beginning to initialize Daemon [[28574,0],2] checking in as pid 43537 on host xserve02.local Daemon [[28574,0],2] not using static ports [saturna.cluster:19337] defining message event: base/plm_base_launch_support.c 423 [xserve02.local:43537] [[28574,0],2] orted: up and running - waiting for commands! Daemon [[28574,0],1] checking in as pid 44056 on host xserve01.cluster Daemon [[28574,0],1] not using static ports [saturna.cluster:19337] defining message event: base/plm_base_launch_support.c 423 [xserve01.cluster:44056] [[28574,0],1] orted: up and running - waiting for commands! [saturna.cluster:19337] defining message event: grpcomm_bad_module.c 183 [saturna.cluster:19337] progressed_wait: base/plm_base_launch_support.c 712 [saturna.cluster:19337] [[28574,0],0] orte:daemon:cmd:processor called by [[28574,0],0] for tag 1 [saturna.cluster:19337] [[28574,0],0] node[0].name saturna daemon 0 arch ffc90200 [saturna.cluster:19337] [[28574,0],0] node[1].name xserve01 daemon 1 arch ffc90200 [saturna.cluster:19337] [[28574,0],0] node[2].name xserve02 daemon 2 arch ffc90200 [saturna.cluster:19337] [[28574,0],0] orted_cmd: received add_local_procs [saturna.cluster:19337] defining message event: base/odls_base_default_fns.c 1219 [saturna.cluster:19337] [[28574,0],0] orte:daemon:send_relay [saturna.cluster:19337] [[28574,0],0] orte:daemon:send_relay sending relay msg to 1 [saturna.cluster:19337] [[28574,0],0] orte:daemon:send_relay sending relay msg to 2 [xserve01.cluster:44056] [[28574,0],1] orted_recv_cmd: received message from [[28574,0],0] [xserve01.cluster:44056] defining message event: orted/orted_comm.c 159 [xserve01.cluster:44056] [[28574,0],1] orted_recv_cmd: reissued recv [xserve01.cluster:44056] [[28574,0],1] orte:daemon:cmd:processor called by [[28574,0],0] for tag 1 [xserve01.cluster:44056] [[28574,0],1] node[0].name saturna daemon 0 arch ffc90200 [xserve01.cluster:44056] [[28574,0],1] node[1].name xserve01 daemon 1 arch ffc90200 [xserve01.cluster:44056] [[28574,0],1] node[2].name xserve02 daemon 2 arch ffc90200 [xserve01.cluster:44056] [[28574,0],1] orted_cmd: received add_local_procs [[28574,0],0] [xserve02.local:43537] defining message event: orted/orted_comm.c 159 [xserve02.local:43537] [[28574,0],2] orted_recv_cmd: reissued recv [xserve02.local:43537] [[28574,0],2] orte:daemon:cmd:processor called by [[28574,0],0] for tag 1 [xserve02.local:43537] [[28574,0],2] node[0].name saturna daemon 0 arch ffc90200 [xserve02.local:43537] [[28574,0],2] node[1].name xserve01 daemon 1 arch ffc90200 [xserve02.local:43537] [[28574,0],2] node[2].name xserve02 daemon 2 arch ffc90200 [xserve02.local:43537] [[28574,0],2] orted_cmd: received add_local_procs [xserve02.local:43537] [[28574,0],2] orte:daemon:send_relay [xserve02.local:43537] [[28574,0],2] orte:daemon:send_relay - recipient list is empty! serve01.cluster:44056] [[28574,0],1] orte:daemon:send_relay - recipient list is empty! [saturna.cluster:19337] defining message event: iof_hnp_receive.c 227 [saturna.cluster:19337] defining message event: base/plm_base_launch_support.c 668 [saturna.cluster:19337] defining message event: iof_hnp_receive.c 227 [saturna.cluster:19337] defining message event: iof_hnp_receive.c 227 MPI init [saturna.cluster:19337] defining message event: iof_hnp_receive.c 227 [saturna.cluster:19337] defining message event: iof_hnp_receive.c 227 MPI init MPI init [saturna.cluster:19337] defining message event: iof_hnp_receive.c 227 MPI init MPI init MPI init [saturna.cluster:19337] defining message event: iof_hnp_receive.c 227 [saturna.cluster:19337] defining message event: iof_hnp_receive.c 227 MPI init MPI init [xserve01.cluster:44056] [[28574,0],1] orted_recv_cmd: received message from [[28574,1],0] [xserve01.cluster:44056] defining message event: orted/orted_comm.c 159 [xserve01.cluster:44056] [[28574,0],1] orted_recv_cmd: reissued recv [xserve01.cluster:44056] [[28574,0],1] orte:daemon:cmd:processor called by [[28574,1],0] for tag 1 [xserve01.cluster:44056] [[28574,0],1] orted_recv: received sync+nidmap from local proc [[28574,1],0] [xserve01.cluster:44056] [[28574,0],1] orte:daemon:cmd:processor: processing commands completed [xserve02.local:43537] [[28574,0],2] orted_recv_cmd: received message from [[28574,1],1] [xserve02.local:43537] defining message event: orted/orted_comm.c 159 [xserve02.local:43537] [[28574,0],2] orted_recv_cmd: reissued recv [xserve02.local:43537] [[28574,0],2] orte:daemon:cmd:processor called by [[28574,1],1] for tag 1 [xserve02.local:43537] [[28574,0],2] orted_recv: received sync+nidmap from local proc [[28574,1],1] [xserve02.local:43537] [[28574,0],2] orte:daemon:cmd:processor: processing commands completed [xserve01.cluster:44056] [[28574,0],1] orted_recv_cmd: received message from [[28574,1],2] [xserve01.cluster:44056] defining message event: orted/orted_comm.c 159 [xserve01.cluster:44056] [[28574,0],1] orted_recv_cmd: reissued recv [xserve01.cluster:44056] [[28574,0],1] orte:daemon:cmd:processor called by [[28574,1],2] for tag 1 [xserve01.cluster:44056] [[28574,0],1] orted_recv: received sync+nidmap from local proc [[28574,1],2] [xserve01.cluster:44056] [[28574,0],1] orte:daemon:cmd:processor: processing commands completed [xserve01.cluster:44056] [[28574,0],1] orted_recv_cmd: received message from [[28574,1],6] [xserve01.cluster:44056] defining message event: orted/orted_comm.c 159 [xserve01.cluster:44056] [[28574,0],1] orted_recv_cmd: reissued recv [xserve01.cluster:44056] [[28574,0],1] orte:daemon:cmd:processor called by [[28574,1],6] for tag 1 [xserve01.cluster:44056] [[28574,0],1] orted_recv: received sync+nidmap from local proc [[28574,1],6] [xserve01.cluster:44056] [[28574,0],1] orte:daemon:cmd:processor: processing commands completed [xserve01.cluster:44056] [[28574,0],1] orted_recv_cmd: received message from [[28574,1],4] [xserve01.cluster:44056] defining message event: orted/orted_comm.c 159 [xserve01.cluster:44056] [[28574,0],1] orted_recv_cmd: reissued recv [xserve01.cluster:44056] [[28574,0],1] orte:daemon:cmd:processor called by [[28574,1],4] for tag 1 [xserve01.cluster:44056] [[28574,0],1] orted_recv: received sync+nidmap from local proc [[28574,1],4] [saturna.cluster:19337] defining message event: base/routed_base_receive.c 153 mmands completed [xserve01.cluster:44056] [[28574,0],1] orted_recv_cmd: received message from [[28574,1],0] [xserve01.cluster:44056] defining message event: orted/orted_comm.c 159 [xserve01.cluster:44056] [[28574,0],1] orted_recv_cmd: reissued recv [xserve01.cluster:44056] [[28574,0],1] orte:daemon:cmd:processor called by [[28574,1],0] for tag 1 [xserve01.cluster:44056] [[28574,0],1] orted_cmd: received collective data cmd [xserve01.cluster:44056] [[28574,0],1] orte:daemon:cmd:processor: processing commands completed [xserve01.cluster:44056] [[28574,0],1] orted_recv_cmd: received message from [[28574,1],6] [xserve01.cluster:44056] defining message event: orted/orted_comm.c 159 [xserve01.cluster:44056] [[28574,0],1] orted_recv_cmd: reissued recv [xserve01.cluster:44056] [[28574,0],1] orte:daemon:cmd:processor called by [[28574,1],6] for tag 1 [xserve01.cluster:44056] [[28574,0],1] orted_cmd: received collective data cmd [xserve01.cluster:44056] [[28574,0],1] orte:daemon:cmd:processor: processing commands completed [xserve01.cluster:44056] [[28574,0],1] orted_recv_cmd: received message from [[28574,1],2] [xserve01.cluster:44056] defining message event: orted/orted_comm.c 159 [xserve01.cluster:44056] [[28574,0],1] orted_recv_cmd: reissued recv [xserve01.cluster:44056] [[28574,0],1] orte:daemon:cmd:processor called by [[28574,1],2] for tag 1 [xserve01.cluster:44056] [[28574,0],1] orted_cmd: received collective data cmd [xserve01.cluster:44056] [[28574,0],1] orte:daemon:cmd:processor: processing commands completed [saturna.cluster:19337] [[28574,0],0] orted_recv_cmd: received message from [[28574,0],1] [xserve01.cluster:44056] defining message event: orted/orted_comm.c 159 [xserve01.cluster:44056] [[28574,0],1] orted_recv_cmd: reissued recv [xserve01.cluster:44056] [[28574,0],1] orte:daemon:cmd:processor called by [[28574,1],4] f[saturna.cluster:19337] defining message event: orted/orted_comm.c 159 lective data cmd [xserve01.cluster:44056] [[28574,0],1] orte:daemon:cmd:processor: processing commands completed [saturna.cluster:19337] [[28574,0],0] orted_recv_cmd: reissued recv [saturna.cluster:19337] [[28574,0],0] orte:daemon:cmd:processor called by [[28574,0],1] for tag 1 [saturna.cluster:19337] [[28574,0],0] orted_cmd: received collective data cmd [saturna.cluster:19337] [[28574,0],0] orte:daemon:cmd:processor: processing commands completed [xserve02.local:43537] [[28574,0],2] orted_recv_cmd: received message from [[28574,1],3] [xserve02.local:43537] defining message event: orted/orted_comm.c 159 [xserve02.local:43537] [[28574,0],2] orted_recv_cmd: reissued recv [xserve02.local:43537] [[28574,0],2] orte:daemon:cmd:processor called by [[28574,1],3] for tag 1 [xserve02.local:43537] [[28574,0],2] orted_recv: received sync+nidmap from local proc [[28574,1],3] [xserve02.local:43537] [[28574,0],2] orte:daemon:cmd:processor: processing commands completed [xserve02.local:43537] [[28574,0],2] orted_recv_cmd: received message from [[28574,1],7] [xserve02.local:43537] defining message event: orted/orted_comm.c 159 [xserve02.local:43537] [[28574,0],2] orted_recv_cmd: reissued recv [xserve02.local:43537] [[28574,0],2] orte:daemon:cmd:processor called by [[28574,1],7] for tag 1 [xserve02.local:43537] [[28574,0],2] orted_recv: received sync+nidmap from local proc [[28574,1],7] [xserve02.local:43537] [[28574,0],2] orte:daemon:cmd:processor: processing commands completed [xserve02.local:43537] [[28574,0],2] orted_recv_cmd: received message from [[28574,1],5] [xserve02.local:43537] defining message event: orted/orted_comm.c 159 [xserve02.local:43537] [[28574,0],2] orted_recv_cmd: reissued recv [xserve02.local:43537] [[28574,0],2] orte:daemon:cmd:processor called by [[28574,1],5] for tag 1 [xserve02.local:43537] [[28574,0],2] orted_recv: received sync+nidmap from local proc [[28574,1],5] [xserve02.local:43537] [[28574,0],2] orte:daemon:cmd:processor: processing commands completed [xserve02.local:43537] [[28574,0],2] orted_recv_cmd: received message from [[28574,1],1] [xserve02.local:43537] defining message event: orted/orted_comm.c 159 [xserve02.local:43537] [[28574,0],2] orted_recv_cmd: reissued recv [xserve02.local:43537] [[28574,0],2] orte:daemon:cmd:processor called by [[28574,1],1] for tag 1 [xserve02.local:43537] [[28574,0],2] orted_cmd: received collective data cmd [xserve02.local:43537] [[28574,0],2] orte:daemon:cmd:processor: processing commands completed [xserve02.local:43537] [[28574,0],2] orted_recv_cmd: received message from [[28574,1],5] [xserve02.local:43537] defining message event: orted/orted_comm.c 159 [xserve02.local:43537] [[28574,0],2] orted_recv_cmd: reissued recv [xserve02.local:43537] [[28574,0],2] orte:daemon:cmd:processor called by [[28574,1],5] for tag 1 [xserve02.local:43537] [[28574,0],2] orted_cmd: received collective data cmd [xserve02.local:43537] [[28574,0],2] orte:daemon:cmd:processor: processing commands completed [xserve02.local:43537] [[28574,0],2] orted_recv_cmd: received message from [[28574,1],3] [xserve02.local:43537] defining message event: orted/orted_comm.c 159 [xserve02.local:43537] [[28574,0],2] orted_recv_cmd: reissued recv [xserve02.local:43537] [[28574,0],2] orte:daemon:cmd:processor called by [[28574,1],3] for tag 1 [xserve02.local:43537] [[28574,0],2] orted_cmd: received collective data cmd [xserve02.local:43537] [[28574,0],2] orte:daemon:cmd:processor: processing commands completed [xserve02.local:43537] [[28574,0],2] orted_recv_cmd: received message from [[28574,1],7] [xserve02.local:43537] defining message event: orted/orted_comm.c 159 [x[saturna.cluster:19337] [[28574,0],0] orted_recv_cmd: reissued recv erve02.local:43537] [[28574,0],2] orte:daemon:cmd:processor called by [[28574,1],7] for tag 1 [xserve02.local:43537] [[28574,0],2] orted_cmd: received collective data cmd [xserve02.local:43537] [[28574,0],2] orte:daemon:cmd:processor: processing commands completed [saturna.cluster:19337] [[28574,0],0] orte:daemon:cmd:processor called by [[28574,0],2] for tag 1 [saturna.cluster:19337] [[28574,0],0] orted_cmd: received collective data cmd [saturna.cluster:19337] defining message event: grpcomm_bad_module.c 183 [saturna.cluster:19337] [[28574,0],0] orte:daemon:cmd:processor: processing commands completed [saturna.cluster:19337] [[28574,0],0] orte:daemon:cmd:processor called by [[28574,0],0] for tag 1 [saturna.cluster:19337] [[28574,0],0] orted_cmd: received message_local_procs [saturna.cluster:19337] [[28574,0],0] orted:comm:message_local_procs delivering message to job [28574,1] tag 15 [saturna.cluster:19337] [[28574,0],0] orte:daemon:send_relay [saturna.cluster:19337] [[28574,0],0] orte:daemon:send_relay sending relay msg to 1 [saturna.cluster:19337] [[28574,0],0] orte:daemon:send_relay sending relay msg to 2 [xserve02.local:43537] [[28574,0],2] orted_recv_cmd: received message from [[28574,0],0] [xserve02.local:43537] defining message event: orted/orted_comm.c 159 [xserve02.local:43537] [[28574,0],2] orted_recv_cmd: reissued recv [xserve02.local:43537] [[28574,0],2] orte:daemon:cmd:processor called by [[28574,0],0] for tag 1 [xserve02.local:43537] [[28574,0],2] orted_cmd: received message_local_procs [xserve02.local:43537] [[28574,0],2] orted:comm:message_local_procs delivering message to job [28574,1] tag 15 [xserve02.local:43537] [[28574,0],2] orte:daemon:send_relay [xserve02.local:43537] [[28574,0],2] orte:daemon:send_relay - recipient list is empty! [xserve01.cluster:44056] [[28574,0],1] orted_recv_cmd: received message from [[28574,0],0] [xserve01.cluster:44056] defining message event: orted/orted_comm.c 159 [xserve01.cluster:44056] [[28574,0],1] orted_recv_cmd: reissued recv [xserve01.cluster:44056] [[28574,0],1] orte:daemon:cmd:processor called by [[28574,0],0] for tag 1 [xserve01.cluster:44056] [[28574,0],1] orted_cmd: received message_local_procs [xserve01.cluster:44056] [[28574,0],1] orted:comm:message_local_procs delivering message to job [28574,1] tag 15 [xserve01.cluster:44056] [[28574,0],1] orte:daemon:send_relay [xserve01.cluster:44056] [[28574,0],1] orte:daemon:send_relay - recipient list is empty! [xserve02.local:43537] [[28574,0],2] orted_recv_cmd: received message from [[28574,1],5] [xserve02.local:43537] defining message event: orted/orted_comm.c 159 [xserve02.local:43537] [[28574,0],2] orted_recv_cmd: reissued recv [xserve02.local:43537] [[28574,0],2] orte:daemon:cmd:processor called by [[28574,1],5] for tag 1 [xserve02.local:43537] [[28574,0],2] orted_cmd: received collective data cmd [xserve02.local:43537] [[28574,0],2] orte:daemon:cmd:processor: processing commands completed [xserve02.local:43537] [[28574,0],2] orted_recv_cmd: received message from [[28574,1],3] [xserve02.local:43537] defining message event: orted/orted_comm.c 159 [xserve02.local:43537] [[28574,0],2] orted_recv_cmd: reissued recv [xserve02.local:43537] [[28574,0],2] orte:daemon:cmd:processor called by [[28574,1],3] for tag 1 [xserve02.local:43537] [[28574,0],2] orted_cmd: received collective data cmd [xserve02.local:43537] [[28574,0],2] orte:daemon:cmd:processor: processing commands completed [xserve02.local:43537] [[28574,0],2] orted_recv_cmd: received message from [[28574,1],7] [xserve02.local:43537] defining message event: orted/orted_comm.c 159 [xserve02.local:43537] [[28574,0],2] orted_recv_cmd: reissued recv [xserve02.local:43537] [[28574,0],2] orte:daemon:cmd:processor called by [[28574,1],7] for tag 1 [xserve02.local:43537] [[28574,0],2] orted_cmd: received collective data cmd [xserve02.local:43537] [[28574,0],2] orte:daemon:cmd:processor: processing commands completed [xserve01.cluster:44056] [[28574,0],1] orted_recv_cmd: received message from [[28574,1],0] [xserve01.cluster:44056] defining message event: orted/orted_comm.c 159 [xserve01.cluster:44056] [[28574,0],1] orted_recv_cmd: reissued recv [xserve01.cluster:44056] [[28574,0],1] orte:daemon:cmd:processor called by [[28574,1],0] for tag 1 [xserve01.cluster:44056] [[28574,0],1] orted_cmd: received collective data cmd [xserve01.cluster:44056] [[28574,0],1] orte:daemon:cmd:processor: processing commands completed [xserve01.cluster:44056] [[28574,0],1] orted_recv_cmd: received message from [[28574,1],6] [xserve01.cluster:44056] defining message event: orted/orted_comm.c 159 [xserve01.cluster:44056] [[28574,0],1] orted_recv_cmd: reissued recv [xserve01.cluster:44056] [[28574,0],1] orte:daemon:cmd:processor called by [[28574,1],6] for tag 1 [xserve01.cluster:44056] [[28574,0],1] orted_cmd: received collective data cmd [xserve01.cluster:44056] [[28574,0],1] orte:daemon:cmd:processor: processing commands completed [xserve01.cluster:44056] [[28574,0],1] orted_recv_cmd: received message from [[28574,1],2] [xserve01.cluster:44056] defining message event: orted/orted_comm.c 159 [xserve01.cluster:44056] [[28574,0],1] orted_recv_cmd: reissued recv [xserve01.cluster:44056] [[28574,0],1] orte:daemon:cmd:processor called by [[28574,1],2] for tag 1 [xserve01.cluster:44056] [[28574,0],1] orted_cmd: received collective data cmd [xserve01.cluster:44056] [[28574,0],1] orte:daemon:cmd:processor: processing commands completed [xserve01.cluster:44056] [[28574,0],1] orted_recv_cmd: received message from [[28574,1],4] [xserve01.cluster:44056] defining message event: orted/orted_comm.c 159 [xserve01.cluster:44056] [[28574,0],1] orted_recv_cmd: reissued recv [xserve01.cluster:44056] [[28574,0],1] orte:daemon:cmd:processor called by [[28574,1],4] for tag 1 [xserve01.cluster:44056] [[28574,0],1] orted_cmd: received col[saturna.cluster:19337] [[28574,0],0] orted_recv_cmd: reissued recv cmd:processor: processing commands completed [saturna.cluster:19337] [[28574,0],0] orte:daemon:cmd:processor called by [[28574,0],1] for tag 1 [saturna.cluster:19337] [[28574,0],0] orted_cmd: received collective data cmd [saturna.cluster:19337] [[28574,0],0] orte:daemon:cmd:processor: processing commands completed [xserve02.local:43537] [[28574,0],2] orted_recv_cmd: received message from [[28574,1],1] [xserve02.local:43537] defining message event: orted/orted_comm.c 159 [x[saturna.cluster:19337] [[28574,0],0] orted_recv_cmd: reissued recv erve02.local:43537] [[28574,0],2] orte:daemon:cmd:processor called by [[28574,1],1] for tag 1 [xserve02.local:43537] [[28574,0],2] orted_cmd: received collective data cmd [xserve02.local:43537] [[28574,0],2] orte:daemon:cmd:processor: processing commands completed [saturna.cluster:19337] [[28574,0],0] orte:daemon:cmd:processor called by [[28574,0],2] for tag 1 [saturna.cluster:19337] [[28574,0],0] orted_cmd: received collective data cmd [saturna.cluster:19337] defining message event: grpcomm_bad_module.c 183 [saturna.cluster:19337] [[28574,0],0] orte:daemon:cmd:processor: processing commands completed [saturna.cluster:19337] [[28574,0],0] orte:daemon:cmd:processor called by [[28574,0],0] for tag 1 [saturna.cluster:19337] [[28574,0],0] orted_cmd: received message_local_procs [saturna.cluster:19337] [[28574,0],0] orted:comm:message_local_procs delivering message to job [28574,1] tag 17 [saturna.cluster:19337] [[28574,0],0] orte:daemon:send_relay [saturna.cluster:19337] [[28574,0],0] orte:daemon:send_relay sending relay msg to 1 [saturna.cluster:19337] [[28574,0],0] orte:daemon:send_relay sending relay msg to 2 [xserve02.local:43537] [[28574,0],2] orted_recv_cmd: received message from [[28574,0],0] [xserve02.local:43537] defining message event: orted/orted_comm.c 159 [xserve02.local:43537] [[28574,0],2] orted_recv_cmd: reissued recv [xserve02.local:43537] [[28574,0],2] orte:daemon:cmd:processor called by [[28574,0],0] for tag 1 [xserve02.local:43537] [[28574,0],2] orted_cmd: received message_local_procs [xserve02.local:43537] [[28574,0],2] orted:comm:message_local_procs delivering message to job [28574,1] tag 17 [xserve01.cluster:44056] [[28574,0],1] orted_recv_cmd: received message from [[28574,0],0] [xserve01.cluster:44056] defining message event: orted/orted_comm.c 159 [xserve01.cluster:44056] [[28574,0],1] orted_recv_cmd: reissued recv [xserve01.cluster:44056] [[28574,0],1] orte:daemon:cmd:processor called by [[28574,0],0] for tag 1 [xserve01.cluster:44056] [[28574,0],1] orted_cmd: received message_local_procs [xserve01.cluster:44056] [[28574,0],1] orted:comm:message_local_procs delivering message to job [28574,1] tag 17 [xserve02.local:43537] [[28574,0],2] orte:daemon:send_relay [xserve02.local:43537] [[28574,0],2] orte:daemon:send_relay - recipient list is empty! ty! [saturna.cluster:19337] defining message event: iof_hnp_receive.c 227 [saturna.cluster:19337] defining message event: iof_hnp_receive.c 227 [saturna.cluster:19337] defining message event: iof_hnp_receive.c 227 Done MPI init checking connection between rank 0 on xserve01.cluster and rank 1 [saturna.cluster:19337] defining message event: iof_hnp_receive.c 227 Done MPI init Done MPI init Done MPI init [saturna.cluster:19337] defining message event: iof_hnp_receive.c 227 [saturna.cluster:19337] defining message event: iof_hnp_receive.c 227 [saturna.cluster:19337] defining message event: iof_hnp_receive.c 227 Done MPI init [saturna.cluster:19337] defining message event: iof_hnp_receive.c 227 Done MPI init Done MPI init Done MPI init [saturna.cluster:19337] defining message event: iof_hnp_receive.c 227 checking connection between rank 1 on xserve02.local and rank 2 [saturna.cluster:19337] defining message event: iof_hnp_receive.c 227 checking connection between rank 0 on xserve01.cluster and rank 2 checking connection between rank 0 on xserve01.cluster and rank 3 [saturna.cluster:19337] defining message event: iof_hnp_receive.c 227 checking connection between rank 2 on xserve01.cluster and rank 3 [saturna.cluster:19337] defining message event: iof_hnp_receive.c 227 checking connection between rank 1 on xserve02.local and rank 3 checking connection between rank 1 on xserve02.local and rank 4 [saturna.cluster:19337] defining message event: iof_hnp_receive.c 227 [saturna.cluster:19337] defining message event: iof_hnp_receive.c 227 checking connection between rank 0 on xserve01.cluster and rank 4 checking connection between rank 0 on xserve01.cluster and rank 5 [saturna.cluster:19337] defining message event: iof_hnp_receive.c 227 [saturna.cluster:19337] defining message event: iof_hnp_receive.c 227 checking connection between rank 3 on xserve02.local and rank 4 checking connection between rank 2 on xserve01.cluster and rank 4 checking connection between rank 2 on xserve01.cluster and rank 5 [saturna.cluster:19337] defining message event: iof_hnp_receive.c 227 checking connection between rank 1 on xserve02.local and rank 5 [saturna.cluster:19337] defining message event: iof_hnp_receive.c 227 checking connection between rank 4 on xserve01.cluster and rank 5 [saturna.cluster:19337] defining message event: iof_hnp_receive.c 227 checking connection between rank 3 on xserve02.local and rank 5 Killed by signal 2. 7] defining timer event: 0 sec 0 usec at orterun.c:1128 mpirun: killing job... [saturna.cluster:19337] [[28574,0],0]:orterun.c(1031) updating exit status to 1 [saturna.cluster:19337] defining message event: base/plm_base_orted_cmds.c 276 [saturna.cluster:19337] defining timeout: 0 sec 2000 usec at base/plm_base_orted_cmds.c:321 [saturna.cluster:19337] progressed_wait: base/plm_base_orted_cmds.c 324 [saturna.cluster:19337] defining timeout: 0 sec 8000 usec at orterun.c:1066 [saturna.cluster:19337] [[28574,0],0] orte:daemon:cmd:processor called by [[28574,0],0] for tag 1 [saturna.cluster:19337] defining message event: base/odls_base_default_fns.c 2267 [saturna.cluster:19337] [[28574,0],0] orte:daemon:cmd:processor: processing commands completed [saturna.cluster:19337] [[28574,0],0] calling orted_exit trigger -------------------------------------------------------------------------- mpirun was unable to cleanly terminate the daemons on the nodes shown below. Additional manual cleanup may be required - please refer to the "orte-clean" tool for assistance. -------------------------------------------------------------------------- xserve01 xserve02