On Tue, Aug 11, 2009 at 8:54 AM, Klymak Jody
<jklymak@uvic.ca> wrote:
On 11-Aug-09, at 6:16 AM, Jeff Squyres wrote:
This means that OMPI is finding an mca_iof_proxy.la file at run time from a prior version of Open MPI. You might want to use "find" or "locate" to search your nodes and find it. I suspect that you somehow have an OMPI 1.3.x install that overlaid an install of a prior OMPI version installation.
However, after reinstalling on the nodes (but not cleaning out /usr/lib on all the nodes) I still have the following:
Thanks, Jody
saturna.cluster:17660] mca:base:select:( plm) Querying component [rsh]
[saturna.cluster:17660] mca:base:select:( plm) Query of component [rsh] set priority to 10
[saturna.cluster:17660] mca:base:select:( plm) Querying component [slurm]
[saturna.cluster:17660] mca:base:select:( plm) Skipping component [slurm]. Query failed to return a module
[saturna.cluster:17660] mca:base:select:( plm) Querying component [tm]
[saturna.cluster:17660] mca:base:select:( plm) Skipping component [tm]. Query failed to return a module
[saturna.cluster:17660] mca:base:select:( plm) Querying component [xgrid]
[saturna.cluster:17660] mca:base:select:( plm) Skipping component [xgrid]. Query failed to return a module
[saturna.cluster:17660] mca:base:select:( plm) Selected component [rsh]
[saturna.cluster:17660] plm:base:set_hnp_name: initial bias 17660 nodename hash 1656374957
[saturna.cluster:17660] plm:base:set_hnp_name: final jobfam 24811
[saturna.cluster:17660] [[24811,0],0] plm:base:receive start comm
[saturna.cluster:17660] mca:base:select:( odls) Querying component [default]
[saturna.cluster:17660] mca:base:select:( odls) Query of component [default] set priority to 1
[saturna.cluster:17660] mca:base:select:( odls) Selected component [default]
[saturna.cluster:17660] [[24811,0],0] plm:rsh: setting up job [24811,1]
[saturna.cluster:17660] [[24811,0],0] plm:base:setup_job for job [24811,1]
[saturna.cluster:17660] [[24811,0],0] plm:rsh: local shell: 0 (bash)
[saturna.cluster:17660] [[24811,0],0] plm:rsh: assuming same remote shell as local shell
[saturna.cluster:17660] [[24811,0],0] plm:rsh: remote shell: 0 (bash)
[saturna.cluster:17660] [[24811,0],0] plm:rsh: final template argv:
/usr/bin/ssh <template> PATH=/usr/local/openmpi/bin:$PATH ; export PATH ; LD_LIBRARY_PATH=/usr/local/openmpi/lib:$LD_LIBRARY_PATH ; export LD_LIBRARY_PATH ; /usr/local/openmpi/bin/orted --debug-daemons -mca ess env -mca orte_ess_jobid 1626013696 -mca orte_ess_vpid <template> -mca orte_ess_num_procs 3 --hnp-uri "1626013696.0;tcp://142.104.154.96:49710;tcp://192.168.2.254:49710" -mca plm_base_verbose 5 -mca odls_base_verbose 5
[saturna.cluster:17660] [[24811,0],0] plm:rsh: launching on node xserve01
[saturna.cluster:17660] [[24811,0],0] plm:rsh: recording launch of daemon [[24811,0],1]
[saturna.cluster:17660] [[24811,0],0] plm:rsh: executing: (//usr/bin/ssh) [/usr/bin/ssh xserve01 PATH=/usr/local/openmpi/bin:$PATH ; export PATH ; LD_LIBRARY_PATH=/usr/local/openmpi/lib:$LD_LIBRARY_PATH ; export LD_LIBRARY_PATH ; /usr/local/openmpi/bin/orted --debug-daemons -mca ess env -mca orte_ess_jobid 1626013696 -mca orte_ess_vpid 1 -mca orte_ess_num_procs 3 --hnp-uri "1626013696.0;tcp://142.104.154.96:49710;tcp://192.168.2.254:49710" -mca plm_base_verbose 5 -mca odls_base_verbose 5]
Daemon was launched on xserve01.cluster - beginning to initialize
[xserve01.cluster:42519] mca:base:select:( odls) Querying component [default]
[xserve01.cluster:42519] mca:base:select:( odls) Query of component [default] set priority to 1
[xserve01.cluster:42519] mca:base:select:( odls) Selected component [default]
Daemon [[24811,0],1] checking in as pid 42519 on host xserve01.cluster
Daemon [[24811,0],1] not using static ports
[saturna.cluster:17660] [[24811,0],0] plm:rsh: launching on node xserve02
[saturna.cluster:17660] [[24811,0],0] plm:rsh: recording launch of daemon [[24811,0],2]
[saturna.cluster:17660] [[24811,0],0] plm:rsh: executing: (//usr/bin/ssh) [/usr/bin/ssh xserve02 PATH=/usr/local/openmpi/bin:$PATH ; export PATH ; LD_LIBRARY_PATH=/usr/local/openmpi/lib:$LD_LIBRARY_PATH ; export LD_LIBRARY_PATH ; /usr/local/openmpi/bin/orted --debug-daemons -mca ess env -mca orte_ess_jobid 1626013696 -mca orte_ess_vpid 2 -mca orte_ess_num_procs 3 --hnp-uri "1626013696.0;tcp://142.104.154.96:49710;tcp://192.168.2.254:49710" -mca plm_base_verbose 5 -mca odls_base_verbose 5]
Daemon was launched on xserve02.local - beginning to initialize
[xserve02.local:42180] mca:base:select:( odls) Querying component [default]
[xserve02.local:42180] mca:base:select:( odls) Query of component [default] set priority to 1
[xserve02.local:42180] mca:base:select:( odls) Selected component [default]
Daemon [[24811,0],2] checking in as pid 42180 on host xserve02.local
Daemon [[24811,0],2] not using static ports
[saturna.cluster:17660] [[24811,0],0] plm:base:daemon_callback
[saturna.cluster:17660] progressed_wait: base/plm_base_launch_support.c 459
[saturna.cluster:17660] defining message event: base/plm_base_launch_support.c 423
[saturna.cluster:17660] [[24811,0],0] plm:base:orted_report_launch from daemon [[24811,0],1]
[saturna.cluster:17660] [[24811,0],0] plm:base:orted_report_launch completed for daemon [[24811,0],1]
[saturna.cluster:17660] defining message event: base/plm_base_launch_support.c 423
[saturna.cluster:17660] [[24811,0],0] plm:base:orted_report_launch from daemon [[24811,0],2]
[xserve01.cluster:42519] [[24811,0],1] orted: up and running - waiting for commands!
[saturna.cluster:17660] [[24811,0],0] plm:base:orted_report_launch completed for daemon [[24811,0],2]
[saturna.cluster:17660] [[24811,0],0] plm:base:daemon_callback completed
[saturna.cluster:17660] [[24811,0],0] plm:base:launch_apps for job [24811,1]
[xserve02.local:42180] [[24811,0],2] orted: up and running - waiting for commands!
[saturna.cluster:17660] defining message event: grpcomm_bad_module.c 183
[saturna.cluster:17660] [[24811,0],0] plm:base:report_launched for job [24811,1]
[saturna.cluster:17660] progressed_wait: base/plm_base_launch_support.c 712
[saturna.cluster:17660] [[24811,0],0] orte:daemon:cmd:processor called by [[24811,0],0] for tag 1
[saturna.cluster:17660] [[24811,0],0] node[0].name saturna daemon 0 arch ffc90200
[saturna.cluster:17660] [[24811,0],0] node[1].name xserve01 daemon 1 arch ffc90200
[saturna.cluster:17660] [[24811,0],0] node[2].name xserve02 daemon 2 arch ffc90200
[saturna.cluster:17660] [[24811,0],0] orted_cmd: received add_local_procs
[saturna.cluster:17660] [[24811,0],0] odls:constructing child list
[saturna.cluster:17660] [[24811,0],0] odls:construct_child_list unpacking data to launch job [24811,1]
[saturna.cluster:17660] [[24811,0],0] odls:construct_child_list adding new jobdat for job [24811,1]
[saturna.cluster:17660] [[24811,0],0] odls:construct_child_list unpacking 1 app_contexts
[saturna.cluster:17660] [[24811,0],0] odls:constructing child list - checking proc 0 on node 1 with daemon 1
[saturna.cluster:17660] [[24811,0],0] odls:constructing child list - checking proc 1 on node 2 with daemon 2
[saturna.cluster:17660] [[24811,0],0] odls:constructing child list - checking proc 2 on node 1 with daemon 1
[saturna.cluster:17660] [[24811,0],0] odls:constructing child list - checking proc 3 on node 2 with daemon 2
[saturna.cluster:17660] [[24811,0],0] odls:constructing child list - checking proc 4 on node 1 with daemon 1
[saturna.cluster:17660] [[24811,0],0] odls:constructing child list - checking proc 5 on node 2 with daemon 2
[saturna.cluster:17660] [[24811,0],0] odls:constructing child list - checking proc 6 on node 1 with daemon 1
[saturna.cluster:17660] [[24811,0],0] odls:constructing child list - checking proc 7 on node 2 with daemon 2
[saturna.cluster:17660] [[24811,0],0] odls:constructing child list - checking proc 8 on node 1 with daemon 1
[saturna.cluster:17660] [[24811,0],0] odls:constructing child list - checking proc 9 on node 2 with daemon 2
[saturna.cluster:17660] [[24811,0],0] odls:constructing child list - checking proc 10 on node 1 with daemon 1
[saturna.cluster:17660] [[24811,0],0] odls:constructing child list - checking proc 11 on node 2 with daemon 2
[saturna.cluster:17660] [[24811,0],0] odls:constructing child list - checking proc 12 on node 1 with daemon 1
[saturna.cluster:17660] [[24811,0],0] odls:constructing child list - checking proc 13 on node 2 with daemon 2
[saturna.cluster:17660] [[24811,0],0] odls:constructing child list - checking proc 14 on node 1 with daemon 1
[saturna.cluster:17660] [[24811,0],0] odls:constructing child list - checking proc 15 on node 2 with daemon 2
[saturna.cluster:17660] [[24811,0],0] odls:construct:child: num_participating 2
[saturna.cluster:17660] [[24811,0],0] odls:launch found 4 processors for 0 children and set oversubscribed to false
[saturna.cluster:17660] [[24811,0],0] odls:launch reporting job [24811,1] launch status
[saturna.cluster:17660] defining message event: base/odls_base_default_fns.c 1219
[saturna.cluster:17660] [[24811,0],0] odls:launch setting waitpids
[saturna.cluster:17660] [[24811,0],0] orte:daemon:send_relay
[saturna.cluster:17660] [[24811,0],0] orte:daemon:send_relay sending relay msg to 1
[saturna.cluster:17660] [[24811,0],0] orte:daemon:send_relay sending relay msg to 2
[saturna.cluster:17660] [[24811,0],0] plm:base:app_report_launch from daemon [[24811,0],0]
[saturna.cluster:17660] [[24811,0],0] plm:base:app_report_launch completed processing
[xserve01.cluster:42519] [[24811,0],1] orted_recv_cmd: received message from [[24811,0],0]
[xserve01.cluster:42519] defining message event: orted/orted_comm.c 159
[xserve01.cluster:42519] [[24811,0],1] orted_recv_cmd: reissued recv
[xserve01.cluster:42519] [[24811,0],1] orte:daemon:cmd:processor called by [[24811,0],0] for tag 1
[xserve01.cluster:42519] [[24811,0],1] node[0].name saturna daemon 0 arch ffc90200
[xserve01.cluster:42519] [[24811,0],1] node[1].name xserve01 daemon 1 arch ffc90200
[xserve01.cluster:42519] [[24811,0],1] node[2].name xserve02 daemon 2 arch ffc90200
[xserve01.cluster:42519] [[24811,0],1] orted_cmd: received add_local_procs
[xserve01.cluster:42519] [[24811,0],1] odls:constructing child list
[xserve01.cluster:42519] [[24811,0],1] odls:construct_child_list unpacking data to launch job [24811,1]
[xserve01.cluster:42519] [[24811,0],1] odls:construct_child_list adding new jobdat for job [24811,1]
[xserve01.cluster:42519] [[24811,0],1] odls:construct_child_list unpacking 1 app_contexts
[xserve02.local:42180] [[24811,0],2] orted_recv_cmd: received message from [[24811,0],0]
[xserve02.local:42180] defining message event: orted/orted_comm.c 159
[xserve02.local:42180] [[24811,0],2] orted_recv_cmd: reissued recv
[xserve02.local:42180] [[24811,0],2] orte:daemon:cmd:processor called by [[24811,0],0] for tag 1
[xserve02.local:42180] [[24811,0],2] node[0].name saturna daemon 0 arch ffc90200
[xserve02.local:42180] [[24811,0],2] node[1].name xserve01 daemon 1 arch ffc90200
[xserve02.local:42180] [[24811,0],2] node[2].name xserve02 daemon 2 arch ffc90200
[xserve02.local:42180] [[24811,0],2] orted_cmd: received add_local_procs
[xserve02.local:42180] [[24811,0],2] odls:constructing child list
[xserve02.local:42180] [[24811,0],2] odls:construct_child_list unpacking data to launch job [24811,1]
[xserve02.local:42180] [[24811,0],2] odls:construct_child_list adding new jobdat for job [24811,1]
[xserve01.cluster:42519] [[24811,0],1] odls:constructing child list - checking proc 0 on node 1 with daemon 1
[xserve01.cluster:42519] [[24811,0],1] odls:constructing child list - found proc 0 for me!
[xserve01.cluster:42519] [[24811,0],1] odls:constructing child list - checking proc 1 on node 2 with daemon 2
[xserve01.cluster:42519] [[24811,0],1] odls:constructing child list - checking proc 2 on node 1 with daemon 1
[xserve01.cluster:42519] [[24811,0],1] odls:constructing child list - found proc 2 for me!
[xserve01.cluster:42519] [[24811,0],1] odls:constructing child list - checking proc 3 on node 2 with daemon 2
[xserve01.cluster:42519] [[24811,0],1] odls:constructing child list - checking proc 4 on node 1 with daemon 1
[xserve01.cluster:42519] [[24811,0],1] odls:constructing child list - found proc 4 for me!
[xserve01.cluster:42519] [[24811,0],1] odls:constructing child list - checking proc 5 on node 2 with daemon 2
[xserve01.cluster:42519] [[24811,0],1] odls:constructing child list - checking proc 6 on node 1 with daemon 1
[xserve01.cluster:42519] [[24811,0],1] odls:constructing child list - found proc 6 for me!
[xserve01.cluster:42519] [[24811,0],1] odls:constructing child list - checking proc 7 on node 2 with daemon 2
[xserve01.cluster:42519] [[24811,0],1] odls:constructing child list - checking proc 8 on node 1 with daemon 1
[xserve01.cluster:42519] [[24811,0],1] odls:constructing child list - found proc 8 for me!
[xserve01.cluster:42519] [[24811,0],1] odls:constructing child list - checking proc 9 on node 2 with daemon 2
[xserve01.cluster:42519] [[24811,0],1] odls:constructing child list - checking proc 10 on node 1 with daemon 1
[xserve01.cluster:42519] [[24811,0],1] odls:constructing child list - found proc 10 for me!
[xserve01.cluster:42519] [[24811,0],1] odls:constructing child list - checking proc 11 on node 2 with daemon 2
[xserve01.cluster:42519] [[24811,0],1] odls:constructing child list - checking proc 12 on node 1 with daemon 1
[xserve01.cluster:42519] [[24811,0],1] odls:constructing child list - found proc 12 for me!
[xserve02.local:42180] [[24811,0],2] odls:construct_child_list unpacking 1 app_contexts
[xserve01.cluster:42519] [[24811,0],1] odls:constructing child list - checking proc 13 on node 2 with daemon 2
[xserve01.cluster:42519] [[24811,0],1] odls:constructing child list - checking proc 14 on node 1 with daemon 1
[xserve01.cluster:42519] [[24811,0],1] odls:constructing child list - found proc 14 for me!
[xserve01.cluster:42519] [[24811,0],1] odls:constructing child list - checking proc 15 on node 2 with daemon 2
[xserve01.cluster:42519] [[24811,0],1] odls:construct:child: num_participating 1
[xserve01.cluster:42519] [[24811,0],1] odls:launch found 16 processors for 8 children and set oversubscribed to false
[xserve02.local:42180] [[24811,0],2] odls:constructing child list - checking proc 0 on node 1 with daemon 1
[xserve02.local:42180] [[24811,0],2] odls:constructing child list - checking proc 1 on node 2 with daemon 2
[xserve02.local:42180] [[24811,0],2] odls:constructing child list - found proc 1 for me!
[xserve02.local:42180] [[24811,0],2] odls:constructing child list - checking proc 2 on node 1 with daemon 1
[xserve02.local:42180] [[24811,0],2] odls:constructing child list - checking proc 3 on node 2 with daemon 2
[xserve02.local:42180] [[24811,0],2] odls:constructing child list - found proc 3 for me!
[xserve02.local:42180] [[24811,0],2] odls:constructing child list - checking proc 4 on node 1 with daemon 1
[xserve02.local:42180] [[24811,0],2] odls:constructing child list - checking proc 5 on node 2 with daemon 2
[xserve02.local:42180] [[24811,0],2] odls:constructing child list - found proc 5 for me!
[xserve02.local:42180] [[24811,0],2] odls:constructing child list - checking proc 6 on node 1 with daemon 1
[xserve02.local:42180] [[24811,0],2] odls:constructing child list - checking proc 7 on node 2 with daemon 2
[xserve02.local:42180] [[24811,0],2] odls:constructing child list - found proc 7 for me!
[xserve02.local:42180] [[24811,0],2] odls:constructing child list - checking proc 8 on node 1 with daemon 1
[xserve02.local:42180] [[24811,0],2] odls:constructing child list - checking proc 9 on node 2 with daemon 2
[xserve02.local:42180] [[24811,0],2] odls:constructing child list - found proc 9 for me!
[xserve02.local:42180] [[24811,0],2] odls:constructing child list - checking proc 10 on node 1 with daemon 1
[xserve02.local:42180] [[24811,0],2] odls:constructing child list - checking proc 11 on node 2 with daemon 2
[xserve02.local:42180] [[24811,0],2] odls:constructing child list - found proc 11 for me!
[xserve02.local:42180] [[24811,0],2] odls:constructing child list - checking proc 12 on node 1 with daemon 1
[xserve02.local:42180] [[24811,0],2] odls:constructing child list - checking proc 13 on node 2 with daemon 2
[xserve02.local:42180] [[24811,0],2] odls:constructing child list - found proc 13 for me!
[xserve02.local:42180] [[24811,0],2] odls:constructing child list - checking proc 14 on node 1 with daemon 1
[xserve02.local:42180] [[24811,0],2] odls:constructing child list - checking proc 15 on node 2 with daemon 2
[xserve02.local:42180] [[24811,0],2] odls:constructing child list - found proc 15 for me!
[xserve02.local:42180] [[24811,0],2] odls:construct:child: num_participating 1
[xserve02.local:42180] [[24811,0],2] odls:launch found 16 processors for 8 children and set oversubscribed to false
[xserve01.cluster:42519] [[24811,0],1] odls:launch reporting job [24811,1] launch status
[saturna.cluster:17660] defining message event: base/plm_base_launch_support.c 668
[saturna.cluster:17660] [[24811,0],0] plm:base:app_report_launch reissuing non-blocking recv
[saturna.cluster:17660] [[24811,0],0] plm:base:app_report_launch from daemon [[24811,0],1]
[saturna.cluster:17660] [[24811,0],0] plm:base:app_report_launched for proc [[24811,1],0] from daemon [[24811,0],1]: pid 42523 state 2 exit 0
[saturna.cluster:17660] [[24811,0],0] plm:base:app_report_launched for proc [[24811,1],2] from daemon [[24811,0],1]: pid 42524 state 2 exit 0
[saturna.cluster:17660] [[24811,0],0] plm:base:app_report_launched for proc [[24811,1],4] from daemon [[24811,0],1]: pid 42525 state 2 exit 0
[saturna.cluster:17660] [[24811,0],0] plm:base:app_report_launched for proc [[24811,1],6] from daemon [[24811,0],1]: pid 42526 state 2 exit 0
[xserve01.cluster:42519] [[24811,0],1] odls:launch setting waitpids
[xserve01.cluster:42519] [[24811,0],1] orte:daemon:send_relay
[xserve01.cluster:42519] [[24811,0],1] orte:daemon:send_relay - recipient list is empty!
[saturna.cluster:17660] [[24811,0],0] plm:base:app_report_launched for proc [[24811,1],8] from daemon [[24811,0],1]: pid 42527 state 2 exit 0
[saturna.cluster:17660] [[24811,0],0] plm:base:app_report_launched for proc [[24811,1],10] from daemon [[24811,0],1]: pid 42528 state 2 exit 0
[saturna.cluster:17660] [[24811,0],0] plm:base:app_report_launched for proc [[24811,1],12] from daemon [[24811,0],1]: pid 42529 state 2 exit 0
[saturna.cluster:17660] [[24811,0],0] plm:base:app_report_launched for proc [[24811,1],14] from daemon [[24811,0],1]: pid 42530 state 2 exit 0
[saturna.cluster:17660] [[24811,0],0] plm:base:app_report_launch completed processing
[xserve02.local:42180] [[24811,0],2] odls:launch reporting job [24811,1] launch status
[saturna.cluster:17660] defining message event: base/plm_base_launch_support.c 668
[saturna.cluster:17660] [[24811,0],0] plm:base:app_report_launch reissuing non-blocking recv
[saturna.cluster:17660] [[24811,0],0] plm:base:app_report_launch from daemon [[24811,0],2]
[saturna.cluster:17660] [[24811,0],0] plm:base:app_report_launched for proc [[24811,1],1] from daemon [[24811,0],2]: pid 42184 state 2 exit 0
[saturna.cluster:17660] [[24811,0],0] plm:base:app_report_launched for proc [[24811,1],3] from daemon [[24811,0],2]: pid 42185 state 2 exit 0
[saturna.cluster:17660] [[24811,0],0] plm:base:app_report_launched for proc [[24811,1],5] from daemon [[24811,0],2]: pid 42186 state 2 exit 0
[saturna.cluster:17660] [[24811,0],0] plm:base:app_report_launched for proc [[24811,1],7] from daemon [[24811,0],2]: pid 42187 state 2 exit 0
[saturna.cluster:17660] [[24811,0],0] plm:base:app_report_launched for proc [[24811,1],9] from daemon [[24811,0],2]: pid 42188 state 2 exit 0
[saturna.cluster:17660] [[24811,0],0] plm:base:app_report_launched for proc [[24811,1],11] from daemon [[24811,0],2]: pid 42189 state 2 exit 0
[saturna.cluster:17660] [[24811,0],0] plm:base:app_report_launched for proc [[24811,1],13] from daemon [[24811,0],2]: pid 42190 state 2 exit 0
[saturna.cluster:17660] [[24811,0],0] plm:base:app_report_launched for proc [[24811,1],15] from daemon [[24811,0],2]: pid 42191 state 2 exit 0
[saturna.cluster:17660] [[24811,0],0] plm:base:app_report_launch completed processing
[saturna.cluster:17660] [[24811,0],0] plm:base:report_launched all apps reported
[saturna.cluster:17660] [[24811,0],0] plm:base:launch wiring up iof
[xserve02.local:42180] [[24811,0],2] odls:launch setting waitpids
[xserve02.local:42180] [[24811,0],2] orte:daemon:send_relay
[xserve02.local:42180] [[24811,0],2] orte:daemon:send_relay - recipient list is empty!
[saturna.cluster:17660] [[24811,0],0] plm:base:launch completed for job [24811,1]
[xserve01.cluster:42519] [[24811,0],1] orted_recv_cmd: received message from [[24811,1],0]
[xserve01.cluster:42519] defining message event: orted/orted_comm.c 159
[xserve01.cluster:42519] [[24811,0],1] orted_recv_cmd: reissued recv
[xserve01.cluster:42519] [[24811,0],1] orte:daemon:cmd:processor called by [[24811,1],0] for tag 1
[xserve01.cluster:42519] [[24811,0],1] orted_recv: received sync+nidmap from local proc [[24811,1],0]
[xserve01.cluster:42519] [[24811,0],1] odls: registering sync on child [[24811,1],0]
[xserve01.cluster:42519] [[24811,0],1] odls:sync nidmap requested for job [24811,1]
[xserve01.cluster:42519] [[24811,0],1] odls: sending sync ack to child [[24811,1],0] with 307 bytes of data
[xserve01.cluster:42519] [[24811,0],1] orte:daemon:cmd:processor: processing commands completed
[xserve01.cluster:42519] [[24811,0],1] orted_recv_cmd: received message from [[24811,1],4]
[xserve01.cluster:42519] defining message event: orted/orted_comm.c 159
[xserve01.cluster:42519] [[24811,0],1] orted_recv_cmd: reissued recv
[xserve01.cluster:42519] [[24811,0],1] orte:daemon:cmd:processor called by [[24811,1],4] for tag 1
[xserve01.cluster:42519] [[24811,0],1] orted_recv: received sync+nidmap from local proc [[24811,1],4]
[xserve01.cluster:42519] [[24811,0],1] odls: registering sync on child [[24811,1],4]
[xserve01.cluster:42519] [[24811,0],1] odls:sync nidmap requested for job [24811,1]
[xserve01.cluster:42519] [[24811,0],1] odls: sending sync ack to child [[24811,1],4] with 307 bytes of data
[xserve01.cluster:42519] [[24811,0],1] orte:daemon:cmd:processor: processing commands completed
[xserve01.cluster:42519] [[24811,0],1] orted_recv_cmd: received message from [[24811,1],2]
[xserve01.cluster:42519] defining message event: orted/orted_comm.c 159
[xserve01.cluster:42519] [[24811,0],1] orted_recv_cmd: reissued recv
[xserve01.cluster:42519] [[24811,0],1] orte:daemon:cmd:processor called by [[24811,1],2] for tag 1
[xserve01.cluster:42519] [[24811,0],1] orted_recv: received sync+nidmap from local proc [[24811,1],2]
[xserve01.cluster:42519] [[24811,0],1] odls: registering sync on child [[24811,1],2]
[xserve01.cluster:42519] [[24811,0],1] odls:sync nidmap requested for job [24811,1]
[xserve01.cluster:42519] [[24811,0],1] odls: sending sync ack to child [[24811,1],2] with 307 bytes of data
[xserve01.cluster:42519] [[24811,0],1] orte:daemon:cmd:processor: processing commands completed
[xserve01.cluster:42519] [[24811,0],1] orted_recv_cmd: received message from [[24811,1],6]
[xserve01.cluster:42519] defining message event: orted/orted_comm.c 159
[xserve01.cluster:42519] [[24811,0],1] orted_recv_cmd: reissued recv
[xserve01.cluster:42519] [[24811,0],1] orte:daemon:cmd:processor called by [[24811,1],6] for tag 1
[xserve01.cluster:42519] [[24811,0],1] orted_recv: received sync+nidmap from local proc [[24811,1],6]
[xserve01.cluster:42519] [[24811,0],1] odls: registering sync on child [[24811,1],6]
[xserve01.cluster:42519] [[24811,0],1] odls:sync nidmap requested for job [24811,1]
[xserve01.cluster:42519] [[24811,0],1] odls: sending sync ack to child [[24811,1],6] with 307 bytes of data
[xserve01.cluster:42519] [[24811,0],1] orte:daemon:cmd:processor: processing commands completed
[xserve01.cluster:42519] [[24811,0],1] orted_recv_cmd: received message from [[24811,1],10]
[xserve01.cluster:42519] defining message event: orted/orted_comm.c 159
[xserve01.cluster:42519] [[24811,0],1] orted_recv_cmd: reissued recv
[xserve01.cluster:42519] [[24811,0],1] orte:daemon:cmd:processor called by [[24811,1],10] for tag 1
[xserve01.cluster:42519] [[24811,0],1] orted_recv: received sync+nidmap from local proc [[24811,1],10]
[xserve01.cluster:42519] [[24811,0],1] odls: registering sync on child [[24811,1],10]
[xserve01.cluster:42519] [[24811,0],1] odls:sync nidmap requested for job [24811,1]
[xserve01.cluster:42519] [[24811,0],1] odls: sending sync ack to child [[24811,1],10] with 307 bytes of data
[xserve01.cluster:42519] [[24811,0],1] orte:daemon:cmd:processor: processing commands completed
[xserve01.cluster:42519] [[24811,0],1] orted_recv_cmd: received message from [[24811,1],8]
[xserve01.cluster:42519] defining message event: orted/orted_comm.c 159
[xserve01.cluster:42519] [[24811,0],1] orted_recv_cmd: reissued recv
[xserve01.cluster:42519] [[24811,0],1] orte:daemon:cmd:processor called by [[24811,1],8] for tag 1
[xserve01.cluster:42519] [[24811,0],1] orted_recv: received sync+nidmap from local proc [[24811,1],8]
[xserve01.cluster:42519] [[24811,0],1] odls: registering sync on child [[24811,1],8]
[xserve01.cluster:42519] [[24811,0],1] odls:sync nidmap requested for job [24811,1]
[xserve01.cluster:42519] [[24811,0],1] odls: sending sync ack to child [[24811,1],8] with 307 bytes of data
[xserve01.cluster:42519] [[24811,0],1] orte:daemon:cmd:processor: processing commands completed
[xserve02.local:42180] [[24811,0],2] orted_recv_cmd: received message from [[24811,1],5]
[xserve02.local:42180] defining message event: orted/orted_comm.c 159
[xserve02.local:42180] [[24811,0],2] orted_recv_cmd: reissued recv
[xserve02.local:42180] [[24811,0],2] orte:daemon:cmd:processor called by [[24811,1],5] for tag 1
[xserve02.local:42180] [[24811,0],2] orted_recv: received sync+nidmap from local proc [[24811,1],5]
[xserve02.local:42180] [[24811,0],2] odls: registering sync on child [[24811,1],5]
[xserve02.local:42180] [[24811,0],2] odls:sync nidmap requested for job [24811,1]
[xserve02.local:42180] [[24811,0],2] odls: sending sync ack to child [[24811,1],5] with 307 bytes of data
[xserve02.local:42180] [[24811,0],2] orte:daemon:cmd:processor: processing commands completed
[xserve02.local:42180] [[24811,0],2] orted_recv_cmd: received message from [[24811,1],1]
[xserve02.local:42180] defining message event: orted/orted_comm.c 159
[xserve02.local:42180] [[24811,0],2] orted_recv_cmd: reissued recv
[xserve02.local:42180] [[24811,0],2] orte:daemon:cmd:processor called by [[24811,1],1] for tag 1
[xserve02.local:42180] [[24811,0],2] orted_recv: received sync+nidmap from local proc [[24811,1],1]
[xserve02.local:42180] [[24811,0],2] odls: registering sync on child [[24811,1],1]
[xserve02.local:42180] [[24811,0],2] odls:sync nidmap requested for job [24811,1]
[xserve02.local:42180] [[24811,0],2] odls: sending sync ack to child [[24811,1],1] with 307 bytes of data
[xserve02.local:42180] [[24811,0],2] orte:daemon:cmd:processor: processing commands completed
[xserve02.local:42180] [[24811,0],2] orted_recv_cmd: received message from [[24811,1],3]
[xserve02.local:42180] defining message event: orted/orted_comm.c 159
[xserve02.local:42180] [[24811,0],2] orted_recv_cmd: reissued recv
[xserve02.local:42180] [[24811,0],2] orte:daemon:cmd:processor called by [[24811,1],3] for tag 1
[xserve02.local:42180] [[24811,0],2] orted_recv: received sync+nidmap from local proc [[24811,1],3]
[xserve02.local:42180] [[24811,0],2] odls: registering sync on child [[24811,1],3]
[xserve02.local:42180] [[24811,0],2] odls:sync nidmap requested for job [24811,1]
[xserve02.local:42180] [[24811,0],2] odls: sending sync ack to child [[24811,1],3] with 307 bytes of data
[xserve02.local:42180] [[24811,0],2] orte:daemon:cmd:processor: processing commands completed
[xserve01.cluster:42519] [[24811,0],1] orted_recv_cmd: received message from [[24811,1],12]
[xserve01.cluster:42519] defining message event: orted/orted_comm.c 159
[xserve01.cluster:42519] [[24811,0],1] orted_recv_cmd: reissued recv
[xserve01.cluster:42519] [[24811,0],1] orte:daemon:cmd:processor called by [[24811,1],12] for tag 1
[xserve01.cluster:42519] [[24811,0],1] orted_recv: received sync+nidmap from local proc [[24811,1],12]
[xserve01.cluster:42519] [[24811,0],1] odls: registering sync on child [[24811,1],12]
[xserve01.cluster:42519] [[24811,0],1] odls:sync nidmap requested for job [24811,1]
[xserve01.cluster:42519] [[24811,0],1] odls: sending sync ack to child [[24811,1],12] with 307 bytes of data
[xserve01.cluster:42519] [[24811,0],1] orte:daemon:cmd:processor: processing commands completed
[xserve01.cluster:42519] [[24811,0],1] orted_recv_cmd: received message from [[24811,1],14]
[xserve01.cluster:42519] defining message event: orted/orted_comm.c 159
[xserve01.cluster:42519] [[24811,0],1] orted_recv_cmd: reissued recv
[xserve01.cluster:42519] [[24811,0],1] orte:daemon:cmd:processor called by [[24811,1],14] for tag 1
[xserve01.cluster:42519] [[24811,0],1] orted_recv: received sync+nidmap from local proc [[24811,1],14]
[xserve01.cluster:42519] [[24811,0],1] odls: registering sync on child [[24811,1],14]
[xserve01.cluster:42519] [[24811,0],1] odls:sync nidmap requested for job [24811,1]
[xserve01.cluster:42519] [[24811,0],1] odls: sending sync ack to child [[24811,1],14] with 307 bytes of data
[xserve01.cluster:42519] [[24811,0],1] odls: sending contact info to HNP
[xserve01.cluster:42519] [[24811,0],1] orte:daemon:cmd:processor: processing commands completed
[saturna.cluster:17660] defining message event: base/routed_base_receive.c 153
[xserve02.local:42180] [[24811,0],2] orted_recv_cmd: received message from [[24811,1],11]
[xserve02.local:42180] defining message event: orted/orted_comm.c 159
[xserve02.local:42180] [[24811,0],2] orted_recv_cmd: reissued recv
[xserve02.local:42180] [[24811,0],2] orte:daemon:cmd:processor called by [[24811,1],11] for tag 1
[xserve02.local:42180] [[24811,0],2] orted_recv: received sync+nidmap from local proc [[24811,1],11]
[xserve02.local:42180] [[24811,0],2] odls: registering sync on child [[24811,1],11]
[xserve02.local:42180] [[24811,0],2] odls:sync nidmap requested for job [24811,1]
[xserve02.local:42180] [[24811,0],2] odls: sending sync ack to child [[24811,1],11] with 307 bytes of data
[xserve02.local:42180] [[24811,0],2] orte:daemon:cmd:processor: processing commands completed
[xserve01.cluster:42519] [[24811,0],1] orted_recv_cmd: received message from [[24811,1],2]
[xserve01.cluster:42519] defining message event: orted/orted_comm.c 159
[xserve01.cluster:42519] [[24811,0],1] orted_recv_cmd: reissued recv
[xserve02.local:42180] [[24811,0],2] orted_recv_cmd: received message from [[24811,1],7]
[xserve02.local:42180] defining message event: orted/orted_comm.c 159
[xserve02.local:42180] [[24811,0],2] orted_recv_cmd: reissued recv
[xserve02.local:42180] [[24811,0],2] orte:daemon:cmd:processor called by [[24811,1],7] for tag 1
[xserve02.local:42180] [[24811,0],2] orted_recv: received sync+nidmap from local proc [[24811,1],7]
[xserve02.local:42180] [[24811,0],2] odls: registering sync on child [[24811,1],7]
[xserve02.local:42180] [[24811,0],2] odls:sync nidmap requested for job [24811,1]
[xserve02.local:42180] [[24811,0],2] odls: sending sync ack to child [[24811,1],7] with 307 bytes of data
[xserve02.local:42180] [[24811,0],2] orte:daemon:cmd:processor: processing commands completed
[xserve01.cluster:42519] [[24811,0],1] orte:daemon:cmd:processor called by [[24811,1],2] for tag 1
[xserve01.cluster:42519] [[24811,0],1] orted_cmd: received collective data cmd
[xserve01.cluster:42519] [[24811,0],1] odls: collecting data from child [[24811,1],2]
[xserve01.cluster:42519] [[24811,0],1] orte:daemon:cmd:processor: processing commands completed
[xserve02.local:42180] [[24811,0],2] orted_recv_cmd: received message from [[24811,1],9]
[xserve02.local:42180] defining message event: orted/orted_comm.c 159
[xserve02.local:42180] [[24811,0],2] orted_recv_cmd: reissued recv
[xserve02.local:42180] [[24811,0],2] orte:daemon:cmd:processor called by [[24811,1],9] for tag 1
[xserve02.local:42180] [[24811,0],2] orted_recv: received sync+nidmap from local proc [[24811,1],9]
[xserve02.local:42180] [[24811,0],2] odls: registering sync on child [[24811,1],9]
[xserve02.local:42180] [[24811,0],2] odls:sync nidmap requested for job [24811,1]
[xserve02.local:42180] [[24811,0],2] odls: sending sync ack to child [[24811,1],9] with 307 bytes of data
[xserve02.local:42180] [[24811,0],2] orte:daemon:cmd:processor: processing commands completed
[xserve02.local:42180] [[24811,0],2] orted_recv_cmd: received message from [[24811,1],13]
[xserve02.local:42180] defining message event: orted/orted_comm.c 159
[xserve02.local:42180] [[24811,0],2] orted_recv_cmd: reissued recv
[xserve02.local:42180] [[24811,0],2] orte:daemon:cmd:processor called by [[24811,1],13] for tag 1
[xserve02.local:42180] [[24811,0],2] orted_recv: received sync+nidmap from local proc [[24811,1],13]
[xserve02.local:42180] [[24811,0],2] odls: registering sync on child [[24811,1],13]
[xserve02.local:42180] [[24811,0],2] odls:sync nidmap requested for job [24811,1]
[xserve02.local:42180] [[24811,0],2] odls: sending sync ack to child [[24811,1],13] with 307 bytes of data
[xserve02.local:42180] [[24811,0],2] orte:daemon:cmd:processor: processing commands completed
[xserve01.cluster:42519] [[24811,0],1] orted_recv_cmd: received message from [[24811,1],0]
[xserve01.cluster:42519] defining message event: orted/orted_comm.c 159
[xserve01.cluster:42519] [[24811,0],1] orted_recv_cmd: reissued recv
[xserve01.cluster:42519] [[24811,0],1] orte:daemon:cmd:processor called by [[24811,1],0] for tag 1
[xserve01.cluster:42519] [[24811,0],1] orted_cmd: received collective data cmd
[xserve01.cluster:42519] [[24811,0],1] odls: collecting data from child [[24811,1],0]
[xserve01.cluster:42519] [[24811,0],1] orte:daemon:cmd:processor: processing commands completed
[xserve02.local:42180] [[24811,0],2] orted_recv_cmd: received message from [[24811,1],15]
[xserve02.local:42180] defining message event: orted/orted_comm.c 159
[xserve02.local:42180] [[24811,0],2] orted_recv_cmd: reissued recv
[xserve02.local:42180] [[24811,0],2] orte:daemon:cmd:processor called by [[24811,1],15] for tag 1
[xserve02.local:42180] [[24811,0],2] orted_recv: received sync+nidmap from local proc [[24811,1],15]
[xserve02.local:42180] [[24811,0],2] odls: registering sync on child [[24811,1],15]
[xserve02.local:42180] [[24811,0],2] odls:sync nidmap requested for job [24811,1]
[xserve02.local:42180] [[24811,0],2] odls: sending sync ack to child [[24811,1],15] with 307 bytes of data
[xserve02.local:42180] [[24811,0],2] odls: sending contact info to HNP
[xserve02.local:42180] [[24811,0],2] orte:daemon:cmd:processor: processing commands completed
[saturna.cluster:17660] defining message event: base/routed_base_receive.c 153
[xserve01.cluster:42519] [[24811,0],1] orted_recv_cmd: received message from [[24811,1],4]
[xserve01.cluster:42519] defining message event: orted/orted_comm.c 159
[xserve01.cluster:42519] [[24811,0],1] orted_recv_cmd: reissued recv
[xserve01.cluster:42519] [[24811,0],1] orte:daemon:cmd:processor called by [[24811,1],4] for tag 1
[xserve01.cluster:42519] [[24811,0],1] orted_cmd: received collective data cmd
[xserve01.cluster:42519] [[24811,0],1] odls: collecting data from child [[24811,1],4]
[xserve01.cluster:42519] [[24811,0],1] orte:daemon:cmd:processor: processing commands completed
[xserve01.cluster:42519] [[24811,0],1] orted_recv_cmd: received message from [[24811,1],6]
[xserve01.cluster:42519] defining message event: orted/orted_comm.c 159
[xserve01.cluster:42519] [[24811,0],1] orted_recv_cmd: reissued recv
[xserve01.cluster:42519] [[24811,0],1] orte:daemon:cmd:processor called by [[24811,1],6] for tag 1
[xserve01.cluster:42519] [[24811,0],1] orted_cmd: received collective data cmd
[xserve01.cluster:42519] [[24811,0],1] odls: collecting data from child [[24811,1],6]
[xserve01.cluster:42519] [[24811,0],1] orte:daemon:cmd:processor: processing commands completed
[xserve01.cluster:42519] [[24811,0],1] orted_recv_cmd: received message from [[24811,1],10]
[xserve01.cluster:42519] defining message event: orted/orted_comm.c 159
[xserve01.cluster:42519] [[24811,0],1] orted_recv_cmd: reissued recv
[xserve01.cluster:42519] [[24811,0],1] orte:daemon:cmd:processor called by [[24811,1],10] for tag 1
[xserve01.cluster:42519] [[24811,0],1] orted_cmd: received collective data cmd
[xserve01.cluster:42519] [[24811,0],1] odls: collecting data from child [[24811,1],10]
[xserve01.cluster:42519] [[24811,0],1] orte:daemon:cmd:processor: processing commands completed
[xserve01.cluster:42519] [[24811,0],1] orted_recv_cmd: received message from [[24811,1],8]
[xserve01.cluster:42519] defining message event: orted/orted_comm.c 159
[xserve01.cluster:42519] [[24811,0],1] orted_recv_cmd: reissued recv
[xserve01.cluster:42519] [[24811,0],1] orte:daemon:cmd:processor called by [[24811,1],8] for tag 1
[xserve01.cluster:42519] [[24811,0],1] orted_cmd: received collective data cmd
[xserve01.cluster:42519] [[24811,0],1] odls: collecting data from child [[24811,1],8]
[xserve01.cluster:42519] [[24811,0],1] orte:daemon:cmd:processor: processing commands completed
[xserve02.local:42180] [[24811,0],2] orted_recv_cmd: received message from [[24811,1],5]
[xserve02.local:42180] defining message event: orted/orted_comm.c 159
[xserve02.local:42180] [[24811,0],2] orted_recv_cmd: reissued recv
[xserve02.local:42180] [[24811,0],2] orte:daemon:cmd:processor called by [[24811,1],5] for tag 1
[xserve02.local:42180] [[24811,0],2] orted_cmd: received collective data cmd
[xserve02.local:42180] [[24811,0],2] odls: collecting data from child [[24811,1],5]
[xserve02.local:42180] [[24811,0],2] orte:daemon:cmd:processor: processing commands completed
[xserve02.local:42180] [[24811,0],2] orted_recv_cmd: received message from [[24811,1],3]
[xserve02.local:42180] defining message event: orted/orted_comm.c 159
[xserve02.local:42180] [[24811,0],2] orted_recv_cmd: reissued recv
[xserve02.local:42180] [[24811,0],2] orte:daemon:cmd:processor called by [[24811,1],3] for tag 1
[xserve02.local:42180] [[24811,0],2] orted_cmd: received collective data cmd
[xserve02.local:42180] [[24811,0],2] odls: collecting data from child [[24811,1],3]
[xserve02.local:42180] [[24811,0],2] orte:daemon:cmd:processor: processing commands completed
[xserve01.cluster:42519] [[24811,0],1] orted_recv_cmd: received message from [[24811,1],12]
[xserve01.cluster:42519] defining message event: orted/orted_comm.c 159
[xserve01.cluster:42519] [[24811,0],1] orted_recv_cmd: reissued recv
[xserve01.cluster:42519] [[24811,0],1] orte:daemon:cmd:processor called by [[24811,1],12] for tag 1
[xserve01.cluster:42519] [[24811,0],1] orted_cmd: received collective data cmd
[xserve01.cluster:42519] [[24811,0],1] odls: collecting data from child [[24811,1],12]
[xserve01.cluster:42519] [[24811,0],1] orte:daemon:cmd:processor: processing commands completed
[xserve02.local:42180] [[24811,0],2] orted_recv_cmd: received message from [[24811,1],1]
[xserve02.local:42180] defining message event: orted/orted_comm.c 159
[xserve02.local:42180] [[24811,0],2] orted_recv_cmd: reissued recv
[xserve02.local:42180] [[24811,0],2] orte:daemon:cmd:processor called by [[24811,1],1] for tag 1
[xserve02.local:42180] [[24811,0],2] orted_cmd: received collective data cmd
[xserve02.local:42180] [[24811,0],2] odls: collecting data from child [[24811,1],1]
[xserve02.local:42180] [[24811,0],2] orte:daemon:cmd:processor: processing commands completed
[saturna.cluster:17660] [[24811,0],0] orted_recv_cmd: received message from [[24811,0],1]
[saturna.cluster:17660] defining message event: orted/orted_comm.c 159
[xserve01.cluster:42519] [[24811,0],1] orted_recv_cmd: received message from [[24811,1],14]
[xserve01.cluster:42519] defining message event: orted/orted_comm.c 159
[xserve01.cluster:42519] [[24811,0],1] orted_recv_cmd: reissued recv
[xserve01.cluster:42519] [[24811,0],1] orte:daemon:cmd:processor called by [[24811,1],14] for tag 1
[xserve01.cluster:42519] [[24811,0],1] orted_cmd: received collective data cmd
[xserve01.cluster:42519] [[24811,0],1] odls: collecting data from child [[24811,1],14]
[xserve01.cluster:42519] [[24811,0],1] odls: executing collective
[xserve01.cluster:42519] [[24811,0],1] odls: daemon collective called
[xserve01.cluster:42519] [[24811,0],1] odls: daemon collective for job [24811,1] from [[24811,0],1] type 2 num_collected 1 num_participating 1 num_contributors 8
[xserve01.cluster:42519] [[24811,0],1] odls: daemon col[saturna.cluster:17660] [[24811,0],0] orted_recv_cmd: reissued recv
lective not the HNP - sending to parent [[24811,0],0]
[saturna.cluster:17660] [[24811,0],0] orte:daemon:cmd:processor called by [[24811,0],1] for tag 1
[xserve01.cluster:42519] [[24811,0],1] odls: collective completed
[xserve01.cluster:42519] [[24811,0],1] orte:daemon:cmd:processor: processing commands completed
[saturna.cluster:17660] [[24811,0],0] orted_cmd: received collective data cmd
[saturna.cluster:17660] [[24811,0],0] odls: daemon collective called
[saturna.cluster:17660] [[24811,0],0] odls: daemon collective for job [24811,1] from [[24811,0],1] type 2 num_collected 1 num_participating 2 num_contributors 8
[saturna.cluster:17660] [[24811,0],0] orte:daemon:cmd:processor: processing commands completed
[xserve02.local:42180] [[24811,0],2] orted_recv_cmd: received message from [[24811,1],9]
[xserve02.local:42180] defining message event: orted/orted_comm.c 159
[xserve02.local:42180] [[24811,0],2] orted_recv_cmd: reissued recv
[xserve02.local:42180] [[24811,0],2] orte:daemon:cmd:processor called by [[24811,1],9] for tag 1
[xserve02.local:42180] [[24811,0],2] orted_cmd: received collective data cmd
[xserve02.local:42180] [[24811,0],2] odls: collecting data from child [[24811,1],9]
[xserve02.local:42180] [[24811,0],2] orte:daemon:cmd:processor: processing commands completed
[xserve02.local:42180] [[24811,0],2] orted_recv_cmd: received message from [[24811,1],13]
[xserve02.local:42180] defining message event: orted/orted_comm.c 159
[xserve02.local:42180] [[24811,0],2] orted_recv_cmd: reissued recv
[xserve02.local:42180] [[24811,0],2] orte:daemon:cmd:processor called by [[24811,1],13] for tag 1
[xserve02.local:42180] [[24811,0],2] orted_cmd: received collective data cmd
[xserve02.local:42180] [[24811,0],2] odls: collecting data from child [[24811,1],13]
[xserve02.local:42180] [[24811,0],2] orte:daemon:cmd:processor: processing commands completed
[xserve02.local:42180] [[24811,0],2] orted_recv_cmd: received message from [[24811,1],7]
[xserve02.local:42180] defining message event: orted/orted_comm.c 159
[xserve02.local:42180] [[24811,0],2] orted_recv_cmd: reissued recv
[xserve02.local:42180] [[24811,0],2] orte:daemon:cmd:processor called by [[24811,1],7] for tag 1
[xserve02.local:42180] [[24811,0],2] orted_cmd: received collective data cmd
[xserve02.local:42180] [[24811,0],2] odls: collecting data from child [[24811,1],7]
[xserve02.local:42180] [[24811,0],2] orte:daemon:cmd:processor: processing commands completed
[xserve02.local:42180] [[24811,0],2] orted_recv_cmd: received message from [[24811,1],11]
[xserve02.local:42180] defining message event: orted/orted_comm.c 159
[xserve02.local:42180] [[24811,0],2] orted_recv_cmd: reissued recv
[xserve02.local:42180] [[24811,0],2] orte:daemon:cmd:processor called by [[24811,1],11] for tag 1
[xserve02.local:42180] [[24811,0],2] orted_cmd: received collective data cmd
[xserve02.local:42180] [[24811,0],2] odls: collecting data from child [[24811,1],11]
[xserve02.local:42180] [[24811,0],2] orte:daemon:cmd:processor: processing commands completed
[xserve02.local:42180] [[24811,0],2] orted_recv_cmd: received message from [[24811,1],15]
[xserve02.local:42180] defining message event: orted/orted_comm.c 159
[xserve02.local:42180] [[24811,0],2] orted_recv_cmd: reissued recv
[xserve02.local:42180] [[24811,0],2] orte:daemon:cmd:processor called by [[24811,1],15] for tag 1
[xserve02.local:42180] [[24811,0],2] orted_cmd: received collective data cmd
[xserve02.local:42180] [[24811,0],2] odls: collecting data from child [[24811,1],15]
[xserve02.local:42180] [[24811,0],2] odls: executing collective
[xserve02.local:42180] [[24811,0],2] odls: daemon collective called
[xserve02.local:42180] [[24811,0],2] odls: daemon collective for job [24811,1] from [[24811,0],2] type 2 num_collected 1 num_participating 1 num_contributors 8
[xserve02.local:42180] [[24811,0],2] odls: daemon collective not the HNP - sending to parent [[24811,0],0]
[xserve02.local:42180] [[24811,0],2] odls: collective completed
[xserve02.local:42180] [[24811,0],2] orte:daemon:cmd:processor: processing commands completed
[saturna.cluster:17660] [[24811,0],0] orted_recv_cmd: received message from [[24811,0],2]
[saturna.cluster:17660] defining message event: orted/orted_comm.c 159
[saturna.cluster:17660] [[24811,0],0] orted_recv_cmd: reissued recv
[saturna.cluster:17660] [[24811,0],0] orte:daemon:cmd:processor called by [[24811,0],2] for tag 1
[saturna.cluster:17660] [[24811,0],0] orted_cmd: received collective data cmd
[saturna.cluster:17660] [[24811,0],0] odls: daemon collective called
[saturna.cluster:17660] [[24811,0],0] odls: daemon collective for job [24811,1] from [[24811,0],2] type 2 num_collected 2 num_participating 2 num_contributors 16
[saturna.cluster:17660] [[24811,0],0] odls: daemon collective HNP - xcasting to job [24811,1]
[saturna.cluster:17660] defining message event: grpcomm_bad_module.c 183
[saturna.cluster:17660] [[24811,0],0] orte:daemon:cmd:processor: processing commands completed
[saturna.cluster:17660] [[24811,0],0] orte:daemon:cmd:processor called by [[24811,0],0] for tag 1
[saturna.cluster:17660] [[24811,0],0] orted_cmd: received message_local_procs
[saturna.cluster:17660] [[24811,0],0] orted:comm:message_local_procs delivering message to job [24811,1] tag 15
[saturna.cluster:17660] [[24811,0],0] orte:daemon:send_relay
[saturna.cluster:17660] [[24811,0],0] orte:daemon:send_relay sending relay msg to 1
[saturna.cluster:17660] [[24811,0],0] orte:daemon:send_relay sending relay msg to 2
[xserve02.local:42180] [[24811,0],2] orted_recv_cmd: received message from [[24811,0],0]
[xserve02.local:42180] defining message event: orted/orted_comm.c 159
[xserve02.local:42180] [[24811,0],2] orted_recv_cmd: reissued recv
[xserve02.local:42180] [[24811,0],2] orte:daemon:cmd:processor called by [[24811,0],0] for tag 1
[xserve02.local:42180] [[24811,0],2] orted_cmd: received message_local_procs
[xserve02.local:42180] [[24811,0],2] orted:comm:message_local_procs delivering message to job [24811,1] tag 15
[xserve02.local:42180] [[24811,0],2] odls: sending message to tag 15 on child [[24811,1],1]
[xserve02.local:42180] [[24811,0],2] odls: sending message to tag 15 on child [[24811,1],3]
[xserve02.local:42180] [[24811,0],2] odls: sending message to tag 15 on child [[24811,1],5]
[xserve02.local:42180] [[24811,0],2] odls: sending message to tag 15 on child [[24811,1],7]
[xserve02.local:42180] [[24811,0],2] odls: sending message to tag 15 on child [[24811,1],9]
[xserve02.local:42180] [[24811,0],2] odls: sending message to tag 15 on child [[24811,1],11]
[xserve02.local:42180] [[24811,0],2] odls: sending message to tag 15 on child [[24811,1],13]
[xserve01.cluster:42519] [[24811,0],1] orted_recv_cmd: received message from [[24811,0],0]
[xserve01.cluster:42519] defining message event: orted/orted_comm.c 159
[xserve01.cluster:42519] [[24811,0],1] orted_recv_cmd: reissued recv
[xserve01.cluster:42519] [[24811,0],1] orte:daemon:cmd:processor called by [[24811,0],0] for tag 1
[xserve01.cluster:42519] [[24811,0],1] orted_cmd: received message_local_procs
[xserve01.cluster:42519] [[24811,0],1] orted:comm:message_local_procs delivering message to job [24811,1] tag 15
[xserve01.cluster:42519] [[24811,0],1] odls: sending message to tag 15 on child [[24811,1],0]
[xserve01.cluster:42519] [[24811,0],1] odls: sending message to tag 15 on child [[24811,1],2]
[xserve01.cluster:42519] [[24811,0],1] odls: sending message to tag 15 on child [[24811,1],4]
[xserve01.cluster:42519] [[24811,0],1] odls: sending message to tag 15 on child [[24811,1],6]
[xserve01.cluster:42519] [[24811,0],1] odls: sending message to tag 15 on child [[24811,1],8]
[xserve01.cluster:42[xserve02.local:42180] [[24811,0],2] odls: sending message to tag 15 on child [[24811,1],15]
[xserve02.local:42180] [[24811,0],2] orte:daemon:send_relay
[xserve02.local:42180] [[24811,0],2] orte:daemon:send_relay - recipient list is empty!
519] [[24811,0],1] odls: sending message to tag 15 on child [[24811,1],10]
[xserve01.cluster:42519] [[24811,0],1] odls: sending message to tag 15 on child [[24811,1],12]
[xserve01.cluster:42519] [[24811,0],1] odls: sending message to tag 15 on child [[24811,1],14]
[xserve01.cluster:42519] [[24811,0],1] orte:daemon:send_relay
[xserve01.cluster:42519] [[24811,0],1] orte:daemon:send_relay - recipient list is empty!
[xserve02.local:42180] [[24811,0],2] orted_recv_cmd: received message from [[24811,1],5]
[xserve02.local:42180] defining message event: orted/orted_comm.c 159
[xserve02.local:42180] [[24811,0],2] orted_recv_cmd: reissued recv
[xserve02.local:42180] [[24811,0],2] orte:daemon:cmd:processor called by [[24811,1],5] for tag 1
[xserve02.local:42180] [[24811,0],2] orted_cmd: received collective data cmd
[xserve02.local:42180] [[24811,0],2] odls: collecting data from child [[24811,1],5]
[xserve02.local:42180] [[24811,0],2] orte:daemon:cmd:processor: processing commands completed
[xserve02.local:42180] [[24811,0],2] orted_recv_cmd: received message from [[24811,1],13]
[xserve02.local:42180] defining message event: orted/orted_comm.c 159
[xserve02.local:42180] [[24811,0],2] orted_recv_cmd: reissued recv
[xserve02.local:42180] [[24811,0],2] orte:daemon:cmd:processor called by [[24811,1],13] for tag 1
[xserve02.local:42180] [[24811,0],2] orted_cmd: received collective data cmd
[xserve02.local:42180] [[24811,0],2] odls: collecting data from child [[24811,1],13]
[xserve02.local:42180] [[24811,0],2] orte:daemon:cmd:processor: processing commands completed
[xserve02.local:42180] [[24811,0],2] orted_recv_cmd: received message from [[24811,1],7]
[xserve02.local:42180] defining message event: orted/orted_comm.c 159
[xserve02.local:42180] [[24811,0],2] orted_recv_cmd: reissued recv
[xserve02.local:42180] [[24811,0],2] orte:daemon:cmd:processor called by [[24811,1],7] for tag 1
[xserve02.local:42180] [[24811,0],2] orted_cmd: received collective data cmd
[xserve02.local:42180] [[24811,0],2] odls: collecting data from child [[24811,1],7]
[xserve02.local:42180] [[24811,0],2] orte:daemon:cmd:processor: processing commands completed
[xserve02.local:42180] [[24811,0],2] orted_recv_cmd: received message from [[24811,1],9]
[xserve02.local:42180] defining message event: orted/orted_comm.c 159
[xserve02.local:42180] [[24811,0],2] orted_recv_cmd: reissued recv
[xserve02.local:42180] [[24811,0],2] orte:daemon:cmd:processor called by [[24811,1],9] for tag 1
[xserve02.local:42180] [[24811,0],2] orted_cmd: received collective data cmd
[xserve02.local:42180] [[24811,0],2] odls: collecting data from child [[24811,1],9]
[xserve01.cluster:42519] [[24811,0],1] orted_recv_cmd: received message from [[24811,1],12]
[xserve01.cluster:42519] defining message event: orted/orted_comm.c 159
[xserve01.cluster:42519] [[24811,0],1] orted_recv_cmd: reissued recv
[xserve01.cluster:42519] [[24811,0],1] orte:daemon:cmd:processor called by [[24811,1],12] for tag 1
[xserve01.cluster:42519] [[24811,0],1] orted_cmd: received collective data cmd
[xserve02.local:42180] [[24811,0],2] orte:daemon:cmd:processor: processing commands completed
[xserve01.cluster:42519] [[24811,0],1] odls: collecting data from child [[24811,1],12]
[xserve01.cluster:42519] [[24811,0],1] orte:daemon:cmd:processor: processing commands completed
[xserve01.cluster:42519] [[24811,0],1] orted_recv_cmd: received message from [[24811,1],10]
[xserve01.cluster:42519] defining message event: orted/orted_comm.c 159
[xserve01.cluster:42519] [[24811,0],1] orted_recv_cmd: reissued recv
[xserve01.cluster:42519] [[24811,0],1] orte:daemon:cmd:processor called by [[24811,1],10] for tag 1
[xserve01.cluster:42519] [[24811,0],1] orted_cmd: received collective data cmd
[xserve01.cluster:42519] [[24811,0],1] odls: collecting data from child [[24811,1],10]
[xserve01.cluster:42519] [[24811,0],1] orte:daemon:cmd:processor: processing commands completed
[xserve02.local:42180] [[24811,0],2] orted_recv_cmd: received message from [[24811,1],11]
[xserve02.local:42180] defining message event: orted/orted_comm.c 159
[xserve02.local:42180] [[24811,0],2] orted_recv_cmd: reissued recv
[xserve02.local:42180] [[24811,0],2] orte:daemon:cmd:processor called by [[24811,1],11] for tag 1
[xserve02.local:42180] [[24811,0],2] orted_cmd: received collective data cmd
[xserve02.local:42180] [[24811,0],2] odls: collecting data from child [[24811,1],11]
[xserve02.local:42180] [[24811,0],2] orte:daemon:cmd:processor: processing commands completed
[xserve02.local:42180] [[24811,0],2] orted_recv_cmd: received message from [[24811,1],15]
[xserve02.local:42180] defining message event: orted/orted_comm.c 159
[xserve02.local:42180] [[24811,0],2] orted_recv_cmd: reissued recv
[xserve02.local:42180] [[24811,0],2] orte:daemon:cmd:processor called by [[24811,1],15] for tag 1
[xserve02.local:42180] [[24811,0],2] orted_cmd: received collective data cmd
[xserve02.local:42180] [[24811,0],2] odls: collecting data from child [[24811,1],15]
[xserve02.local:42180] [[24811,0],2] orte:daemon:cmd:processor: processing commands completed
[xserve01.cluster:42519] [[24811,0],1] orted_recv_cmd: received message from [[24811,1],4]
[xserve01.cluster:42519] defining message event: orted/orted_comm.c 159
[xserve01.cluster:42519] [[24811,0],1] orted_recv_cmd: reissued recv
[xserve01.cluster:42519] [[24811,0],1] orte:daemon:cmd:processor called by [[24811,1],4] for tag 1
[xserve01.cluster:42519] [[24811,0],1] orted_cmd: received collective data cmd
[xserve01.cluster:42519] [[24811,0],1] odls: collecting data from child [[24811,1],4]
[xserve01.cluster:42519] [[24811,0],1] orte:daemon:cmd:processor: processing commands completed
[xserve01.cluster:42519] [[24811,0],1] orted_recv_cmd: received message from [[24811,1],6]
[xserve01.cluster:42519] defining message event: orted/orted_comm.c 159
[xserve01.cluster:42519] [[24811,0],1] orted_recv_cmd: reissued recv
[xserve01.cluster:42519] [[24811,0],1] orte:daemon:cmd:processor called by [[24811,1],6] for tag 1
[xserve01.cluster:42519] [[24811,0],1] orted_cmd: received collective data cmd
[xserve01.cluster:42519] [[24811,0],1] odls: collecting data from child [[24811,1],6]
[xserve01.cluster:42519] [[24811,0],1] orte:daemon:cmd:processor: processing commands completed
[xserve01.cluster:42519] [[24811,0],1] orted_recv_cmd: received message from [[24811,1],8]
[xserve01.cluster:42519] defining message event: orted/orted_comm.c 159
[xserve01.cluster:42519] [[24811,0],1] orted_recv_cmd: reissued recv
[xserve01.cluster:42519] [[24811,0],1] orte:daemon:cmd:processor called by [[24811,1],8] for tag 1
[xserve01.cluster:42519] [[24811,0],1] orted_cmd: received collective data cmd
[xserve01.cluster:42519] [[24811,0],1] odls: collecting data from child [[24811,1],8]
[xserve01.cluster:42519] [[24811,0],1] orte:daemon:cmd:processor: processing commands completed
[xserve01.cluster:42519] [[24811,0],1] orted_recv_cmd: received message from [[24811,1],14]
[xserve01.cluster:42519] defining message event: orted/orted_comm.c 159
[xserve01.cluster:42519] [[24811,0],1] orted_recv_cmd: reissued recv
[xserve01.cluster:42519] [[24811,0],1] orte:daemon:cmd:processor called by [[24811,1],14] for tag 1
[xserve01.cluster:42519] [[24811,0],1] orted_cmd: received collective data cmd
[xserve01.cluster:42519] [[24811,0],1] odls: collecting data from child [[24811,1],14]
[xserve01.cluster:42519] [[24811,0],1] orte:daemon:cmd:processor: processing commands completed
[xserve02.local:42180] [[24811,0],2] orted_recv_cmd: received message from [[24811,1],1]
[xserve02.local:42180] defining message event: orted/orted_comm.c 159
[xserve02.local:42180] [[24811,0],2] orted_recv_cmd: reissued recv
[xserve02.local:42180] [[24811,0],2] orte:daemon:cmd:processor called by [[24811,1],1] for tag 1
[xserve02.local:42180] [[24811,0],2] orted_cmd: received collective data cmd
[xserve02.local:42180] [[24811,0],2] odls: collecting data from child [[24811,1],1]
[xserve02.local:42180] [[24811,0],2] orte:daemon:cmd:processor: processing commands completed
[xserve02.local:42180] [[24811,0],2] orted_recv_cmd: received message from [[24811,1],3]
[xserve02.local:42180] defining message event: orted/orted_comm.c 159
[xserve02.local:42180] [[24811,0],2] orted_recv_cmd: reissued recv
[xserve02.local:42180] [[24811,0],2] orte:daemon:cmd:processor called by [[24811,1],3] for tag 1
[xserve02.local:42180] [[24811,0],2] orted_cmd: received collective data cmd
[xserve02.local:42180] [[24811,0],2] odls: collecting data from child [[24811,1],3]
[xserve02.local:42180] [[24811,0],2] odls: executing collective
[xserve02.local:42180] [[24811,0],2] odls: daemon collective called
[saturna.cluster:17660] [[24811,0],0] orted_recv_cmd: received message from [[24811,0],2]
[xserve02.local:42180] [[24811,0],2] odls: daemon collective for job [24811,1] from [[24811,0],2] type 1 num_collected 1 num_participating 1 num_contributors 8
[xserve02.local:42180] [[24811,0],2] odls: daemon collective not the HNP - sending to parent [[24811,0],0]
[xserve02.local:42180] [[24811,0],2] odls: collective completed
[xserve02.local:42180] [[24811,0],2] orte:daemon:cmd:processor: processing commands completed
[saturna.cluster:17660] defining message event: orted/orted_comm.c 159
[saturna.cluster:17660] [[24811,0],0] orted_recv_cmd: reissued recv
[saturna.cluster:17660] [[24811,0],0] orted_recv_cmd: received message from [[24811,0],1]
[saturna.cluster:17660] defining message event: orted/orted_comm.c 159
[saturna.cluster:17660] [[24811,0],0] orted_recv_cmd: reissued recv
[xserve01.cluster:42519] [[24811,0],1] orted_recv_cmd: received message from [[24811,1],0]
[xserve01.cluster:42519] defining message event: orted/orted_comm.c 159
[xserve01.cluster:42519] [[24811,0],1] orted_recv_cmd: reissued recv
[xserve01.cluster:42519] [[24811,0],1] orte:daemon:cmd:processor called by [[24811,1],0] for tag 1
[xserve01.cluster:42519] [[24811,0],1] orted_cmd: received collective data cmd
[xserve01.cluster:42519] [[24811,0],1] odls: collecting data from child [[24811,1],0]
[xserve01.cluster:42519] [[24811,0],1] orte:daemon:cmd:processor: processing commands completed
[xserve01.cluster:42519] [[24811,0],1] orted_recv_cmd: received message from [[24811,1],2]
[xserve01.cluster:42519] defining message event: orted/orted_comm.c 159
[xserve01.cluster:42519] [[24811,0],1] orted_recv_cmd: reissued recv
[xserve01.cluster:42519] [[24811,0],1] orte:daemon:cmd:processor called by [[24811,1],2] for tag 1
[xserve01.cluster:42519] [[24811,0],1] orted_cmd: received collective data cmd
[xserve01.cluster:42519] [[24811,0],1] odls: collecting data from child [[24811,1],2]
[xserve01.cluster:42519] [[24811,0],1] odls: executing collective
[xserve01.cluster:42519] [[24811,0],1] odls: daemon collective called
[saturna.cluster:17660] [[24811,0],0] orte:daemon:cmd:processor called by [[24811,0],2] for tag 1
[saturna.cluster:17660] [[24811,0],0] orted_cmd: received collective data cmd
[saturna.cluster:17660] [[24811,0],0] odls: daemon collective called
[saturna.cluster:17660] [[24811,0],0] odls: daemon collective for job [24811,1] from [[24811,0],2] type 1 num_collected 1 num_participating 2 num_contributors 8
[saturna.cluster:17660] [[24811,0],0] orte:daemon:cmd:processor: processing commands completed
[saturna.cluster:17660] [[24811,0],0] orte:daemon:cmd:processor called by [[24811,0],1] for tag 1
[xserve01.cluster:42519] [[24811,0],1] odls: daemon collective for job [24811,1] from [[24811,0],1] type 1 num_collected 1 num_participating 1 num_contributors 8
[xserve01.cluster:42519] [[24811,0],1] odls: daemon collective not the HNP - sending to parent [[24811,0],0]
[xserve01.cluster:42519] [[24811,0],1] odls: collective completed
[xserve01.cluster:42519] [[24811,0],1] orte:daemon:cmd:processor: processing commands completed
[saturna.cluster:17660] [[24811,0],0] orted_cmd: received collective data cmd
[saturna.cluster:17660] [[24811,0],0] odls: daemon collective called
[saturna.cluster:17660] [[24811,0],0] odls: daemon collective for job [24811,1] from [[24811,0],1] type 1 num_collected 2 num_participating 2 num_contributors 16
[saturna.cluster:17660] [[24811,0],0] odls: daemon collective HNP - xcasting to job [24811,1]
[saturna.cluster:17660] defining message event: grpcomm_bad_module.c 183
[saturna.cluster:17660] [[24811,0],0] orte:daemon:cmd:processor: processing commands completed
[saturna.cluster:17660] [[24811,0],0] orte:daemon:cmd:processor called by [[24811,0],0] for tag 1
[saturna.cluster:17660] [[24811,0],0] orted_cmd: received message_local_procs
[saturna.cluster:17660] [[24811,0],0] orted:comm:message_local_procs delivering message to job [24811,1] tag 17
[saturna.cluster:17660] [[24811,0],0] orte:daemon:send_relay
[saturna.cluster:17660] [[24811,0],0] orte:daemon:send_relay sending relay msg to 1
[saturna.cluster:17660] [[24811,0],0] orte:daemon:send_relay sending relay msg to 2
[xserve01.cluster:42519] [[24811,0],1] orted_recv_cmd: received message from [[24811,0],0]
[xserve01.cluster:42519] defining message event: orted/orted_comm.c 159
[xserve01.cluster:42519] [[24811,0],1] orted_recv_cmd: reissued recv
[xserve01.cluster:42519] [[24811,0],1] orte:daemon:cmd:processor called by [[24811,0],0] for tag 1
[xserve01.cluster:42519] [[24811,0],1] orted_cmd: received message_local_procs
[xserve01.cluster:42519] [[24811,0],1] orted:comm:message_local_procs delivering message to job [24811,1] tag 17
[xserve01.cluster:42519] [[24811,0],1] odls: sending message to tag 17 on child [[24811,1],0]
[xserve01.cluster:42519] [[24811,0],1] odls: sending message to tag 17 on child [[24811,1],2]
[xserve01.cluster:42519] [[24811,0],1] odls: sending message to tag 17 on child [[24811,1],4]
[xserve01.cluster:42519] [[24811,0],1] odls: sending message to tag 17 on child [[24811,1],6]
[xserve02.local:42180] [[24811,0],2] orted_recv_cmd: received message from [[24811,0],0]
[xserve02.local:42180] defining message event: orted/orted_comm.c 159
[xserve02.local:42180] [[24811,0],2] orted_recv_cmd: reissued recv
[xserve02.local:42180] [[24811,0],2] orte:daemon:cmd:processor called by [[24811,0],0] for tag 1
[xserve02.local:42180] [[24811,0],2] orted_cmd: received message_local_procs
[xserve02.local:42180] [[24811,0],2] orted:comm:message_local_procs delivering message to job [24811,1] tag 17
[xserve02.local:42180] [[24811,0],2] odls: sending message to tag 17 on child [[24811,1],1]
[xserve02.local:42180] [[24811,0],2] odls: sending message to tag 17 on child [[24811,1],3]
[xserve02.local:42180] [[24811,0],2] odls: sending message to tag 17 on child [[24811,1],5]
[xserve02.local:42180] [[24811,0],2] odls: sending message to tag 17 on child [[24811,1],7]
[xserve02.local:42180] [[24811,0],2] odls: sending message to tag 17 on child [[24811,1],9]
[xserve01.cluster:42519] [[24811,0],1] odls: sending message to tag 17 on child [[24811,1],8]
[xserve01.cluster:42519] [[24811,0],1] odls: sending message to tag 17 on child [[24811,1],10]
[xserve01.cluster:42519] [[24811,0],1] odls: sending message to tag 17 on child [[24811,1],12]
[xserve01.cluster:42519] [[24811,0],1] odls: sending message to tag 17 on child [[24811,1],14]
[xserve01.cluster:42519] [[24811,0],1] orte:daemon:send_relay
[xserve01.cluster:42519] [[24811,0],1] orte:daemon:send_relay - recipient list is empty!
[xserve02.local:42180] [[24811,0],2] odls: sending message to tag 17 on child [[24811,1],11]
[xserve02.local:42180] [[24811,0],2] odls: sending message to tag 17 on child [[24811,1],13]
[xserve02.local:42180] [[24811,0],2] odls: sending message to tag 17 on child [[24811,1],15]
[xserve02.local:42180] [[24811,0],2] orte:daemon:send_relay
[xserve02.local:42180] [[24811,0],2] orte:daemon:send_relay - recipient list is empty!
[saturna.cluster:17660] defining message event: iof_hnp_receive.c 227
[xserve02.local][[24811,1],1][btl_tcp_endpoint.c:486:mca_btl_tcp_endpoint_recv_connect_ack] received unexpected process identifier [[24811,1],2]
[saturna.cluster:17660] defining message event: iof_hnp_receive.c 227
[xserve01.cluster][[24811,1],2][btl_tcp_endpoint.c:486:mca_btl_tcp_endpoint_recv_connect_ack] received unexpected process identifier [[24811,1],5]