[gamma2.censored_domain:22554] connect_uni: connection not allowed
[gamma2.censored_domain:22554] [0,0,0] setting up session dir with
[gamma2.censored_domain:22554] 	universe default-universe-22554
[gamma2.censored_domain:22554] 	user censored_username
[gamma2.censored_domain:22554] 	host gamma2.censored_domain
[gamma2.censored_domain:22554] 	jobid 0
[gamma2.censored_domain:22554] 	procid 0
[gamma2.censored_domain:22554] procdir: /tmp/openmpi-sessions-censored_username@gamma2.censored_domain_0/default-universe-22554/0/0
[gamma2.censored_domain:22554] jobdir: /tmp/openmpi-sessions-censored_username@gamma2.censored_domain_0/default-universe-22554/0
[gamma2.censored_domain:22554] unidir: /tmp/openmpi-sessions-censored_username@gamma2.censored_domain_0/default-universe-22554
[gamma2.censored_domain:22554] top: openmpi-sessions-censored_username@gamma2.censored_domain_0
[gamma2.censored_domain:22554] tmp: /tmp
[gamma2.censored_domain:22554] [0,0,0] contact_file /tmp/openmpi-sessions-censored_username@gamma2.censored_domain_0/default-universe-22554/universe-setup.txt
[gamma2.censored_domain:22554] [0,0,0] wrote setup file
[gamma2.censored_domain:22554] pls:rsh: local csh: 0, local sh: 1
[gamma2.censored_domain:22554] pls:rsh: assuming same remote shell as local shell
[gamma2.censored_domain:22554] pls:rsh: remote csh: 0, remote sh: 1
[gamma2.censored_domain:22554] pls:rsh: final template argv:
[gamma2.censored_domain:22554] pls:rsh:     /usr/bin/ssh <template> orted --debug --bootproxy 1 --name <template> --num_procs 3 --vpid_start 0 --nodename <template> --universe censored_username@gamma2.censored_domain:default-universe-22554 --nsreplica "0.0.0;tcp://129.234.194.162:37450" --gprreplica "0.0.0;tcp://129.234.194.162:37450"
[gamma2.censored_domain:22554] pls:rsh: launching on node gamma2
[gamma2.censored_domain:22554] pls:rsh: gamma2 is a LOCAL node
[gamma2.censored_domain:22554] pls:rsh: reset PATH: /usr/lib/openmpi/1.2.5-gcc/bin:/usr/kerberos/bin:/usr/local/bin:/bin:/usr/bin:/usr/lib/openmpi/1.2.5-gcc/bin:/summary/scripts:/data/dst2/censored_username/dst/dst_server/scripts
[gamma2.censored_domain:22554] pls:rsh: reset LD_LIBRARY_PATH: /usr/lib/openmpi/1.2.5-gcc/lib::/usr/lib/openmpi/1.2.5-gcc/lib
[gamma2.censored_domain:22554] pls:rsh: changing to directory /home/censored_username
[gamma2.censored_domain:22554] pls:rsh: executing: (/usr/lib/openmpi/1.2.5-gcc/bin/orted) orted --debug --bootproxy 1 --name 0.0.1 --num_procs 3 --vpid_start 0 --nodename gamma2 --universe censored_username@gamma2.censored_domain:default-universe-22554 --nsreplica "0.0.0;tcp://129.234.194.162:37450" --gprreplica "0.0.0;tcp://censored_ip:37450" --set-sid [DATADST=/data/dst2/censored_username/dst HOSTNAME=gamma2.censored_domain TERM=xterm-color SHELL=/bin/bash HISTSIZE=1000 KDE_NO_IPV6=1 SSH_CLIENT=censored_client_ip 42057 22 CVSROOT=/home/censored_username/cvsroot OLDPWD=/home/censored_username SSH_TTY=/dev/pts/3 USER=censored_username LD_LIBRARY_PATH=/usr/lib/openmpi/1.2.5-gcc/lib::/usr/lib/openmpi/1.2.5-gcc/lib LS_COLORS=no=00:fi=00:di=01;34:ln=01;36:pi=40;33:so=01;35:bd=40;33;01:cd=40;33;01:or=01;05;37;41:mi=01;05;37;41:ex=01;32:*.cmd=01;32:*.exe=01;32:*.com=01;32:*.btm=01;32:*.bat=01;32:*.sh=01;32:*.csh=01;32:*.tar=01;31:*.tgz=01;31:*.arj=01;31:*.taz=01;31:*.lzh=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.gz=01;31:*.bz2=01;31:*.bz=01;31:*.tz=01;31:*.rpm=01;31:*.cpio=01;31:*.jpg=01;35:*.gif=01;35:*.bmp=01;35:*.xbm=01;35:*.xpm=01;35:*.png=01;35:*.tif=01;35: DST_SERVER=/data/dst2/censored_username/dst/dst_server KDEDIR=/usr DATAXRAY=/data/dst2/censored_username/X-ray MAIL=/var/spool/mail/censored_username PATH=/usr/lib/openmpi/1.2.5-gcc/bin:/usr/kerberos/bin:/usr/local/bin:/bin:/usr/bin:/usr/lib/openmpi/1.2.5-gcc/bin:/summary/scripts:/data/dst2/censored_username/dst/dst_server/scripts INPUTRC=/etc/inputrc PWD=/data/dst2/censored_username/mpitest LANG=en_US.UTF-8 KDE_IS_PRELINKED=1 CALDBCONFIG=/data/dst2/censored_username/X-ray/CALDB/caldb.config DATARAW=/data/dst2/censored_username/X-ray SSH_ASKPASS=/usr/libexec/openssh/gnome-ssh-askpass SHLVL=1 HOME=/home/censored_username LOGNAME=censored_username CVS_RSH=ssh SSH_CONNECTION=censored_client_ip 42057 censored_ip 22 LESSOPEN=|/usr/bin/lesspipe.sh %s CALDB=ftp://heasarc.gsfc.nasa.gov/FTP/caldb DISPLAY=localhost:12.0 CALDBALIAS=/data/dst2/censored_username/X-ray/CALDB/alias_config.fits G_BROKEN_FILENAMES=1 _=/usr/lib/openmpi/1.2.5-gcc/bin/mpirun OMPI_MCA_rds_hostfile_path=hostfile OMPI_MCA_orte_debug=1 OMPI_MCA_btl=tcp,self OMPI_MCA_seed=0]
[gamma2.censored_domain:22554] pls:rsh: launching on node gamma3
[gamma2.censored_domain:22554] pls:rsh: gamma3 is a REMOTE node
[gamma2.censored_domain:22554] pls:rsh: executing: (//usr/bin/ssh) /usr/bin/ssh gamma3  PATH=/usr/lib/openmpi/1.2.5-gcc/bin:$PATH ; export PATH ; LD_LIBRARY_PATH=/usr/lib/openmpi/1.2.5-gcc/lib:$LD_LIBRARY_PATH ; export LD_LIBRARY_PATH ; /usr/lib/openmpi/1.2.5-gcc/bin/orted --debug --bootproxy 1 --name 0.0.2 --num_procs 3 --vpid_start 0 --nodename gamma3 --universe censored_username@gamma2.censored_domain:default-universe-22554 --nsreplica "0.0.0;tcp://censored_ip:37450" --gprreplica "0.0.0;tcp://censored_ip:37450" [DATADST=/data/dst2/censored_username/dst HOSTNAME=gamma2.censored_domain TERM=xterm-color SHELL=/bin/bash HISTSIZE=1000 KDE_NO_IPV6=1 SSH_CLIENT=censored_client_ip 42057 22 CVSROOT=/home/censored_username/cvsroot OLDPWD=/home/censored_username SSH_TTY=/dev/pts/3 USER=censored_username LD_LIBRARY_PATH=:/usr/lib/openmpi/1.2.5-gcc/lib LS_COLORS=no=00:fi=00:di=01;34:ln=01;36:pi=40;33:so=01;35:bd=40;33;01:cd=40;33;01:or=01;05;37;41:mi=01;05;37;41:ex=01;32:*.cmd=01;32:*.exe=01;32:*.com=01;32:*.btm=01;32:*.bat=01;32:*.sh=01;32:*.csh=01;32:*.tar=01;31:*.tgz=01;31:*.arj=01;31:*.taz=01;31:*.lzh=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.gz=01;31:*.bz2=01;31:*.bz=01;31:*.tz=01;31:*.rpm=01;31:*.cpio=01;31:*.jpg=01;35:*.gif=01;35:*.bmp=01;35:*.xbm=01;35:*.xpm=01;35:*.png=01;35:*.tif=01;35: DST_SERVER=/data/dst2/censored_username/dst/dst_server KDEDIR=/usr DATAXRAY=/data/dst2/censored_username/X-ray MAIL=/var/spool/mail/censored_username PATH=/usr/kerberos/bin:/usr/local/bin:/bin:/usr/bin:/usr/lib/openmpi/1.2.5-gcc/bin:/summary/scripts:/data/dst2/censored_username/dst/dst_server/scripts INPUTRC=/etc/inputrc PWD=/data/dst2/censored_username/mpitest LANG=en_US.UTF-8 KDE_IS_PRELINKED=1 CALDBCONFIG=/data/dst2/censored_username/X-ray/CALDB/caldb.config DATARAW=/data/dst2/censored_username/X-ray SSH_ASKPASS=/usr/libexec/openssh/gnome-ssh-askpass SHLVL=1 HOME=/home/censored_username LOGNAME=censored_username CVS_RSH=ssh SSH_CONNECTION=censored_client_ip 42057 censored_ip 22 LESSOPEN=|/usr/bin/lesspipe.sh %s CALDB=ftp://heasarc.gsfc.nasa.gov/FTP/caldb DISPLAY=localhost:12.0 CALDBALIAS=/data/dst2/censored_username/X-ray/CALDB/alias_config.fits G_BROKEN_FILENAMES=1 _=/usr/lib/openmpi/1.2.5-gcc/bin/mpirun OMPI_MCA_rds_hostfile_path=hostfile OMPI_MCA_orte_debug=1 OMPI_MCA_btl=tcp,self OMPI_MCA_seed=0]
[gamma2.censored_domain:22555] [0,0,1] setting up session dir with
[gamma2.censored_domain:22555] 	universe default-universe-22554
[gamma2.censored_domain:22555] 	user censored_username
[gamma2.censored_domain:22555] 	host gamma2
[gamma2.censored_domain:22555] 	jobid 0
[gamma2.censored_domain:22555] 	procid 1
[gamma2.censored_domain:22555] procdir: /tmp/openmpi-sessions-censored_username@gamma2_0/default-universe-22554/0/1
[gamma2.censored_domain:22555] jobdir: /tmp/openmpi-sessions-censored_username@gamma2_0/default-universe-22554/0
[gamma2.censored_domain:22555] unidir: /tmp/openmpi-sessions-censored_username@gamma2_0/default-universe-22554
[gamma2.censored_domain:22555] top: openmpi-sessions-censored_username@gamma2_0
[gamma2.censored_domain:22555] tmp: /tmp
Killed by signal 2.
mpirun: killing job...

[gamma2.censored_domain:22554] sess_dir_finalize: proc session dir not empty - leaving
[gamma2.censored_domain:22554] [0,0,0] ORTE_ERROR_LOG: Timeout in file base/pls_base_orted_cmds.c at line 275
[gamma2.censored_domain:22554] [0,0,0] ORTE_ERROR_LOG: Timeout in file pls_rsh_module.c at line 1166
[gamma2.censored_domain:22554] [0,0,0] ORTE_ERROR_LOG: Timeout in file errmgr_hnp.c at line 90
[gamma2.censored_domain:22554] spawn: in job_state_callback(jobid = 1, state = 0x80)
[gamma2.censored_domain:22554] ERROR: A daemon on node gamma3 failed to start as expected.
[gamma2.censored_domain:22554] ERROR: There may be more information available from
[gamma2.censored_domain:22554] ERROR: the remote shell (see above).
[gamma2.censored_domain:22554] ERROR: The daemon exited unexpectedly with status 255.
--------------------------------------------------------------------------
WARNING: mpirun is in the process of killing a job, but has detected an
interruption (probably control-C).

It is dangerous to interrupt mpirun while it is killing a job (proper
termination may not be guaranteed).  Hit control-C again within 1
second if you really want to kill mpirun immediately.
--------------------------------------------------------------------------
mpirun: forcibly killing job...
--------------------------------------------------------------------------
WARNING: mpirun has exited before it received notification that all
started processes had terminated.  You should double check and ensure
that there are no runaway processes still executing.
--------------------------------------------------------------------------
[gamma2.censored_domain:22555] OOB: Connection to HNP lost
[gamma2.censored_domain:22554] sess_dir_finalize: proc session dir not empty - leaving
[gamma2.censored_domain:22555] sess_dir_finalize: proc session dir not empty - leaving