[csultra06:00526] [0,0,0] setting up session dir with
[csultra06:00526]       universe default-universe
[csultra06:00526]       user manredd
[csultra06:00526]       host csultra06
[csultra06:00526]       jobid 0
[csultra06:00526]       procid 0
[csultra06:00526] procdir:
/tmp/openmpi-sessions-manredd@csultra06_0/default-universe/0/0
[csultra06:00526] jobdir:
/tmp/openmpi-sessions-manredd@csultra06_0/default-universe/0
[csultra06:00526] unidir:
/tmp/openmpi-sessions-manredd@csultra06_0/default-universe
[csultra06:00526] top: openmpi-sessions-manredd@csultra06_0
[csultra06:00526] tmp: /tmp
[csultra06:00526] [0,0,0] contact_file
/tmp/openmpi-sessions-manredd@csultra06_0/default-universe/universe-setup.txt
[csultra06:00526] [0,0,0] wrote setup file
[csultra06:00526] pls:rsh: local csh: 0, local bash: 1
[csultra06:00526] pls:rsh: assuming same remote shell as local shell
[csultra06:00526] pls:rsh: remote csh: 0, remote bash: 1
[csultra06:00526] pls:rsh: final template argv:
[csultra06:00526] pls:rsh:     /bin/ssh <template> orted --debug --bootproxy 1
--name <template> --num_procs 5 --vpid_start 0 --nodename <template>
--universe manredd@csultra06:default-universe --nsreplica
"0.0.0;tcp://193.1.132.62:51629" --gprreplica "0.0.0;tcp://193.1.132.62:51629"
--mpi-call-yield 0
[csultra06:00526] pls:rsh: launching on node csultra08
[csultra06:00526] pls:rsh: not oversubscribed -- setting mpi_yield_when_idle
to 0
[csultra06:00526] pls:rsh: csultra08 is a REMOTE node
[csultra06:00526] pls:rsh: executing: /bin/ssh csultra08 orted --debug
--bootproxy 1 --name 0.0.1 --num_procs 5 --vpid_start 0 --nodename csultra08
--universe manredd@csultra06:default-universe --nsreplica
"0.0.0;tcp://193.1.132.62:51629" --gprreplica "0.0.0;tcp://193.1.132.62:51629"
--mpi-call-yield 0
[csultra06:00526] pls:rsh: launching on node csultra05
[csultra06:00526] pls:rsh: oversubscribed -- setting mpi_yield_when_idle to 1
(1 2)
[csultra06:00526] pls:rsh: csultra05 is a REMOTE node
[csultra06:00526] pls:rsh: executing: /bin/ssh csultra05 orted --debug
--bootproxy 1 --name 0.0.2 --num_procs 5 --vpid_start 0 --nodename csultra05
--universe manredd@csultra06:default-universe --nsreplica
"0.0.0;tcp://193.1.132.62:51629" --gprreplica "0.0.0;tcp://193.1.132.62:51629"
--mpi-call-yield 1
[csultra08:04400] [0,0,1] setting up session dir with
[csultra08:04400]       universe default-universe
[csultra08:04400]       user manredd
[csultra08:04400]       host csultra08
[csultra08:04400]       jobid 0
[csultra08:04400]       procid 1
[csultra08:04400] procdir:
/tmp/openmpi-sessions-manredd@csultra08_0/default-universe/0/1
[csultra08:04400] jobdir:
/tmp/openmpi-sessions-manredd@csultra08_0/default-universe/0
[csultra08:04400] unidir:
/tmp/openmpi-sessions-manredd@csultra08_0/default-universe
[csultra08:04400] top: openmpi-sessions-manredd@csultra08_0
[csultra08:04400] tmp: /tmp
[csultra06:00526] pls:rsh: launching on node csultra02
[csultra06:00526] pls:rsh: oversubscribed -- setting mpi_yield_when_idle to 1
(1 2)
[csultra06:00526] pls:rsh: csultra02 is a REMOTE node
[csultra06:00526] pls:rsh: executing: /bin/ssh csultra02 orted --debug
--bootproxy 1 --name 0.0.3 --num_procs 5 --vpid_start 0 --nodename csultra02
--universe manredd@csultra06:default-universe --nsreplica
"0.0.0;tcp://193.1.132.62:51629" --gprreplica "0.0.0;tcp://193.1.132.62:51629"
--mpi-call-yield 1
[csultra05:02884] [0,0,2] setting up session dir with
[csultra05:02884]       universe default-universe
[csultra05:02884]       user manredd
[csultra05:02884]       host csultra05
[csultra05:02884]       jobid 0
[csultra05:02884]       procid 2
[csultra05:02884] procdir:
/tmp/openmpi-sessions-manredd@csultra05_0/default-universe/0/2
[csultra05:02884] jobdir:
/tmp/openmpi-sessions-manredd@csultra05_0/default-universe/0
[csultra05:02884] unidir:
/tmp/openmpi-sessions-manredd@csultra05_0/default-universe
[csultra05:02884] top: openmpi-sessions-manredd@csultra05_0
[csultra05:02884] tmp: /tmp
[csultra06:00526] pls:rsh: launching on node csultra06
[csultra06:00526] pls:rsh: oversubscribed -- setting mpi_yield_when_idle to 1
(1 2)
[csultra06:00526] pls:rsh: csultra06 is a LOCAL node
[csultra06:00526] pls:rsh: changing to directory /home/cs/manredd
[csultra06:00526] pls:rsh: executing: orted --debug --bootproxy 1 --name 0.0.4
--num_procs 5 --vpid_start 0 --nodename csultra06 --universe
manredd@csultra06:default-universe --nsreplica
"0.0.0;tcp://193.1.132.62:51629" --gprreplica "0.0.0;tcp://193.1.132.62:51629"
--mpi-call-yield 1
[csultra06:00530] [0,0,4] setting up session dir with
[csultra06:00530]       universe default-universe
[csultra06:00530]       user manredd
[csultra06:00530]       host csultra06
[csultra06:00530]       jobid 0
[csultra06:00530]       procid 4
[csultra06:00530] procdir:
/tmp/openmpi-sessions-manredd@csultra06_0/default-universe/0/4
[csultra06:00530] jobdir:
/tmp/openmpi-sessions-manredd@csultra06_0/default-universe/0
[csultra06:00530] unidir:
/tmp/openmpi-sessions-manredd@csultra06_0/default-universe
[csultra06:00530] top: openmpi-sessions-manredd@csultra06_0
[csultra06:00530] tmp: /tmp
[csultra02:28730] [0,0,3] setting up session dir with
[csultra02:28730]       universe default-universe
[csultra02:28730]       user manredd
[csultra02:28730]       host csultra02
[csultra02:28730]       jobid 0
[csultra02:28730]       procid 3
[csultra02:28730] procdir:
/tmp/openmpi-sessions-manredd@csultra02_0/default-universe/0/3
[csultra02:28730] jobdir:
/tmp/openmpi-sessions-manredd@csultra02_0/default-universe/0
[csultra02:28730] unidir:
/tmp/openmpi-sessions-manredd@csultra02_0/default-universe
[csultra02:28730] top: openmpi-sessions-manredd@csultra02_0
[csultra02:28730] tmp: /tmp
[csultra08:04452] [0,1,3] setting up session dir with
[csultra08:04452]       universe default-universe
[csultra08:04452]       user manredd
[csultra08:04452]       host csultra08
[csultra08:04452]       jobid 1
[csultra08:04452]       procid 3
[csultra08:04452] procdir:
/tmp/openmpi-sessions-manredd@csultra08_0/default-universe/1/3
[csultra08:04452] jobdir:
/tmp/openmpi-sessions-manredd@csultra08_0/default-universe/1
[csultra08:04452] unidir:
/tmp/openmpi-sessions-manredd@csultra08_0/default-universe
[csultra08:04452] top: openmpi-sessions-manredd@csultra08_0
[csultra08:04452] tmp: /tmp
[csultra02:28782] [0,1,1] setting up session dir with
[csultra02:28782]       universe default-universe
[csultra02:28782]       user manredd
[csultra02:28782]       host csultra02
[csultra02:28782]       jobid 1
[csultra02:28782]       procid 1
[csultra02:28782] procdir:
/tmp/openmpi-sessions-manredd@csultra02_0/default-universe/1/1
[csultra02:28782] jobdir:
/tmp/openmpi-sessions-manredd@csultra02_0/default-universe/1
[csultra02:28782] unidir:
/tmp/openmpi-sessions-manredd@csultra02_0/default-universe
[csultra02:28782] top: openmpi-sessions-manredd@csultra02_0
[csultra02:28782] tmp: /tmp
[csultra05:02936] [0,1,2] setting up session dir with
[csultra05:02936]       universe default-universe
[csultra05:02936]       user manredd
[csultra05:02936]       host csultra05
[csultra05:02936]       jobid 1
[csultra05:02936]       procid 2
[csultra05:02936] procdir:
/tmp/openmpi-sessions-manredd@csultra05_0/default-universe/1/2
[csultra05:02936] jobdir:
/tmp/openmpi-sessions-manredd@csultra05_0/default-universe/1
[csultra05:02936] unidir:
/tmp/openmpi-sessions-manredd@csultra05_0/default-universe
[csultra05:02936] top: openmpi-sessions-manredd@csultra05_0
[csultra05:02936] tmp: /tmp
[csultra06:00534] [0,1,4] setting up session dir with
[csultra06:00534]       universe default-universe
[csultra06:00534]       user manredd
[csultra06:00534]       host csultra06
[csultra06:00534]       jobid 1
[csultra06:00534]       procid 4
[csultra06:00534] procdir:
/tmp/openmpi-sessions-manredd@csultra06_0/default-universe/1/4
[csultra06:00534] jobdir:
/tmp/openmpi-sessions-manredd@csultra06_0/default-universe/1
[csultra06:00534] unidir:
/tmp/openmpi-sessions-manredd@csultra06_0/default-universe
[csultra06:00534] top: openmpi-sessions-manredd@csultra06_0
[csultra06:00534] tmp: /tmp
[csultra05:02938] [0,1,6] setting up session dir with
[csultra05:02938]       universe default-universe
[csultra05:02938]       user manredd
[csultra05:02938]       host csultra05
[csultra05:02938]       jobid 1
[csultra05:02938]       procid 6
[csultra05:02938] procdir:
/tmp/openmpi-sessions-manredd@csultra05_0/default-universe/1/6
[csultra05:02938] jobdir:
/tmp/openmpi-sessions-manredd@csultra05_0/default-universe/1
[csultra05:02938] unidir:
/tmp/openmpi-sessions-manredd@csultra05_0/default-universe
[csultra02:28784] [0,1,5] setting up session dir with
[csultra05:02938] top: openmpi-sessions-manredd@csultra05_0
[csultra02:28784]       universe default-universe
[csultra05:02938] tmp: /tmp
[csultra02:28784]       user manredd
[csultra02:28784]       host csultra02
[csultra02:28784]       jobid 1
[csultra02:28784]       procid 5
[csultra02:28784] procdir:
/tmp/openmpi-sessions-manredd@csultra02_0/default-universe/1/5
[csultra02:28784] jobdir:
/tmp/openmpi-sessions-manredd@csultra02_0/default-universe/1
[csultra02:28784] unidir:
/tmp/openmpi-sessions-manredd@csultra02_0/default-universe
[csultra02:28784] top: openmpi-sessions-manredd@csultra02_0
[csultra02:28784] tmp: /tmp
[csultra06:00532] [0,1,0] setting up session dir with
[csultra06:00532]       universe default-universe
[csultra06:00532]       user manredd
[csultra06:00532]       host csultra06
[csultra06:00532]       jobid 1
[csultra06:00532]       procid 0
[csultra06:00532] procdir:
/tmp/openmpi-sessions-manredd@csultra06_0/default-universe/1/0
[csultra06:00532] jobdir:
/tmp/openmpi-sessions-manredd@csultra06_0/default-universe/1
[csultra06:00532] unidir:
/tmp/openmpi-sessions-manredd@csultra06_0/default-universe
[csultra06:00532] top: openmpi-sessions-manredd@csultra06_0
[csultra06:00532] tmp: /tmp
[csultra06:00526] spawn: in job_state_callback(jobid = 1, state = 0x4)
[csultra06:00526] Info: Setting up debugger process table for applications
  MPIR_being_debugged = 0
  MPIR_debug_gate = 0
  MPIR_debug_state = 1
  MPIR_acquired_pre_main = 0
  MPIR_i_am_starter = 0
  MPIR_proctable_size = 7
  MPIR_proctable:
    (i, host, exe, pid) = (0, csultra06,
/home/cs/manredd/OpenMPI/openmpi-1.1a1r9260/MPITESTS/mpiinit, 532)
    (i, host, exe, pid) = (1, csultra02,
/home/cs/manredd/OpenMPI/openmpi-1.1a1r9260/MPITESTS/mpiinit, 28782)
    (i, host, exe, pid) = (2, csultra05,
/home/cs/manredd/OpenMPI/openmpi-1.1a1r9260/MPITESTS/mpiinit, 2936)
    (i, host, exe, pid) = (3, csultra08,
/home/cs/manredd/OpenMPI/openmpi-1.1a1r9260/MPITESTS/mpiinit, 4452)
    (i, host, exe, pid) = (4, csultra06,
/home/cs/manredd/OpenMPI/openmpi-1.1a1r9260/MPITESTS/mpiinit, 534)
    (i, host, exe, pid) = (5, csultra02,
/home/cs/manredd/OpenMPI/openmpi-1.1a1r9260/MPITESTS/mpiinit, 28784)
    (i, host, exe, pid) = (6, csultra05,
/home/cs/manredd/OpenMPI/openmpi-1.1a1r9260/MPITESTS/mpiinit, 2938)
[csultra06:00532] [0,1,0] ompi_mpi_init completed
csultra08:Hello world from 3
[csultra05:02936] [0,1,2] ompi_mpi_init completed
csultra06:Hello world from 0
[csultra08:04452] [0,1,3] ompi_mpi_init completed
csultra05:Hello world from 2
[csultra06:00534] [0,1,4] ompi_mpi_init completed
[csultra05:02938] [0,1,6] ompi_mpi_init completed
csultra06:Hello world from 4
csultra02:Hello world from 1
csultra05:Hello world from 6
[csultra02:28782] [0,1,1] ompi_mpi_init completed
csultra02:Hello world from 5
[csultra02:28784] [0,1,5] ompi_mpi_init completed
[csultra06:00530] sess_dir_finalize: proc session dir not empty - leaving
[csultra06:00530] sess_dir_finalize: proc session dir not empty - leaving
[csultra02:28730] sess_dir_finalize: proc session dir not empty - leaving
[csultra05:02884] sess_dir_finalize: proc session dir not empty - leaving
[csultra08:04400] sess_dir_finalize: proc session dir not empty - leaving
[csultra08:04452] sess_dir_finalize: found proc session dir empty - deleting
[csultra08:04452] sess_dir_finalize: found job session dir empty - deleting
[csultra08:04452] sess_dir_finalize: univ session dir not empty - leaving
[csultra05:02936] sess_dir_finalize: found proc session dir empty - deleting
[csultra06:00532] sess_dir_finalize: found proc session dir empty - deleting
[csultra02:28782] sess_dir_finalize: found proc session dir empty - deleting
[csultra02:28782] sess_dir_finalize: job session dir not empty - leaving
[csultra06:00530] orted: job_state_callback(jobid = 1, state =
ORTE_PROC_STATE_TERMINATED)
[csultra06:00530] sess_dir_finalize: job session dir not empty - leaving
[csultra06:00530] sess_dir_finalize: found proc session dir empty - deleting
[csultra06:00530] sess_dir_finalize: job session dir not empty - leaving
[csultra05:02884] sess_dir_finalize: proc session dir not empty - leaving
[csultra02:28730] sess_dir_finalize: proc session dir not empty - leaving