Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

From: Jeff Squyres (jsquyres_at_[hidden])
Date: 2006-07-28 13:05:28


Trolling through some really old messages that never got replies... :-(

The behavior that you are seeing is happening as the result of a really long
discussion among the OMPI developers when we were writing the TCP device.
The problem is that there is ambiguity when connecting peers across TCP in
Open MPI. Specifically, since OMPI can span multiple TCP networks, each MPI
process may be able to use multiple IP addresses to each to each other MPI
process (and vice versa). So we have to try to figure out which IP
addresses can speak to which others.

For example, say that you have a cluster with 16 nodes on a private ethernet
network. One of these nodes doubles as the head node for the cluster and
therefore has 2 ethernet NICs -- one to the external network and one to the
internal cluster network. But since 16 is a nice number, you also want to
use it for computation as well. So when you mpirun spanning all 16 nodes,
OMPI has to figure out to *not* use the external NIC on the head node and
only use the internal NIC.

TCP connections are only made upon demand which is why you only see this
behavior if two processes actually attempt to communicate via MPI (i.e.,
"hello world" with no sending/receiving works fine, but adding the
MPI_SEND/MPI_RECV makes it fail).

We make connections by having all MPI processes exchange their IP
address(es) and port number(s) during MPI_INIT (via a common rendevouz
point, typically mpirun). Then, whenever a connection is requested between
two processes, we apply a small set of rules to all pair combinations of IP
addresses of those processes:

1. If the two IP addresses match after the subnet mask is applied, assume
that they are mutually routable and allow the connection
2. If the two IP addresses are public, assume that they are mutually
routable and allow the connection
3. Otherwise, the connection is disallowed (this is not an error -- we just
disallow this connection on the hope that some other device can be used to
make a connection).

What is happening in your case is that you're falling through to #3 for all
IP address pair combinations and there is no other device that can reach
these processes. Therefore OMPI thinks that it has no channel to reach the
remote process. So it bails (in a horribly non-descriptive way :-( ).

We actually have a very long comment about this in the TCP code and mention
that your scenario (lots of hosts in a single cluster with private addresses
and relatively narrow subnet masks, even though all addresses are, in fact,
routable to each other) is not currently supported -- and that we need to do
something "better". "Better" in this case probably means having a
configuration file that specifies what hosts are mutually routable when the
above rules don't work. Do you have any suggestions on this front?

On 7/5/06 1:15 PM, "Frank Kahle" <openmpi-user_at_[hidden]> wrote:

> users-request_at_[hidden] wrote:
>> A few clarifying questions:
>>
>> What is your netmask on these hosts?
>>
>> Where is the MPI_ALLREDUCE in your app -- right away, or somewhere deep
>> within the application? Can you replicate this with a simple MPI
>> application that essentially calls MPI_INIT, MPI_ALLREDUCE, and
>> MPI_FINALIZE?
>>
>> Can you replicate this with a simple MPI app that does an MPI_SEND /
>> MPI_RECV between two processes on the different subnets?
>>
>> Thanks.
>>
>>
>
> @ Jeff,
>
> netmask 255.255.255.0
>
> Running a simple "hello world" yields no error on each subnet, but
> running "hello world" on both subnets yields the error
>
> [g5dual.3-net:00436] *** An error occurred in MPI_Send
> [g5dual.3-net:00436] *** on communicator MPI_COMM_WORLD
> [g5dual.3-net:00436] *** MPI_ERR_INTERN: internal error
> [g5dual.3-net:00436] *** MPI_ERRORS_ARE_FATAL (goodbye)
>
> Hope this helps!
>
> Frank
>
>
> Just in case you wanna check the source:
> c Fortran example hello_world
> program hello
> include 'mpif.h'
> integer rank, size, ierror, tag, status(MPI_STATUS_SIZE)
> character*12 message
>
> call MPI_INIT(ierror)
> call MPI_COMM_SIZE(MPI_COMM_WORLD, size, ierror)
> call MPI_COMM_RANK(MPI_COMM_WORLD, rank, ierror)
> tag = 100
>
> if (rank .eq. 0) then
> message = 'Hello, world'
> do i=1, size-1
> call MPI_SEND(message, 12, MPI_CHARACTER, i, tag,
> & MPI_COMM_WORLD, ierror)
> enddo
>
> else
> call MPI_RECV(message, 12, MPI_CHARACTER, 0, tag,
> & MPI_COMM_WORLD, status, ierror)
> endif
>
> print*, 'node', rank, ':', message
> call MPI_FINALIZE(ierror)
> end
>
>
> or the full output:
>
> [powerbook:/Network/CFD/hello] motte% mpirun -d -np 5 --hostfile
> ./hostfile /Network/CFD/hello/hello_world
> [powerbook.2-net:00606] [0,0,0] setting up session dir with
> [powerbook.2-net:00606] universe default-universe
> [powerbook.2-net:00606] user motte
> [powerbook.2-net:00606] host powerbook.2-net
> [powerbook.2-net:00606] jobid 0
> [powerbook.2-net:00606] procid 0
> [powerbook.2-net:00606] procdir:
> /tmp/openmpi-sessions-motte_at_powerbook.2-net_0/default-universe/0/0
> [powerbook.2-net:00606] jobdir:
> /tmp/openmpi-sessions-motte_at_powerbook.2-net_0/default-universe/0
> [powerbook.2-net:00606] unidir:
> /tmp/openmpi-sessions-motte_at_powerbook.2-net_0/default-universe
> [powerbook.2-net:00606] top: openmpi-sessions-motte_at_powerbook.2-net_0
> [powerbook.2-net:00606] tmp: /tmp
> [powerbook.2-net:00606] [0,0,0] contact_file
> /tmp/openmpi-sessions-motte_at_powerbook.2-net_0/default-universe/universe-setup.
> txt
> [powerbook.2-net:00606] [0,0,0] wrote setup file
> [powerbook.2-net:00606] pls:rsh: local csh: 1, local bash: 0
> [powerbook.2-net:00606] pls:rsh: assuming same remote shell as local shell
> [powerbook.2-net:00606] pls:rsh: remote csh: 1, remote bash: 0
> [powerbook.2-net:00606] pls:rsh: final template argv:
> [powerbook.2-net:00606] pls:rsh: /usr/bin/ssh <template> orted
> --debug --bootproxy 1 --name <template> --num_procs 6 --vpid_start 0
> --nodename <template> --universe motte_at_powerbook.2-net:default-universe
> --nsreplica "0.0.0;tcp://192.168.2.3:49443" --gprreplica
> "0.0.0;tcp://192.168.2.3:49443" --mpi-call-yield 0
> [powerbook.2-net:00606] pls:rsh: launching on node Powerbook.2-net
> [powerbook.2-net:00606] pls:rsh: not oversubscribed -- setting
> mpi_yield_when_idle to 0
> [powerbook.2-net:00606] pls:rsh: Powerbook.2-net is a LOCAL node
> [powerbook.2-net:00606] pls:rsh: changing to directory /Users/motte
> [powerbook.2-net:00606] pls:rsh: executing: orted --debug --bootproxy 1
> --name 0.0.1 --num_procs 6 --vpid_start 0 --nodename Powerbook.2-net
> --universe motte_at_powerbook.2-net:default-universe --nsreplica
> "0.0.0;tcp://192.168.2.3:49443" --gprreplica
> "0.0.0;tcp://192.168.2.3:49443" --mpi-call-yield 0
> [powerbook.2-net:00607] [0,0,1] setting up session dir with
> [powerbook.2-net:00607] universe default-universe
> [powerbook.2-net:00607] user motte
> [powerbook.2-net:00607] host Powerbook.2-net
> [powerbook.2-net:00607] jobid 0
> [powerbook.2-net:00607] procid 1
> [powerbook.2-net:00607] procdir:
> /tmp/openmpi-sessions-motte_at_Powerbook.2-net_0/default-universe/0/1
> [powerbook.2-net:00607] jobdir:
> /tmp/openmpi-sessions-motte_at_Powerbook.2-net_0/default-universe/0
> [powerbook.2-net:00607] unidir:
> /tmp/openmpi-sessions-motte_at_Powerbook.2-net_0/default-universe
> [powerbook.2-net:00607] top: openmpi-sessions-motte_at_Powerbook.2-net_0
> [powerbook.2-net:00607] tmp: /tmp
> [powerbook.2-net:00606] pls:rsh: launching on node g4d003.3-net
> [powerbook.2-net:00606] pls:rsh: not oversubscribed -- setting
> mpi_yield_when_idle to 0
> [powerbook.2-net:00606] pls:rsh: g4d003.3-net is a REMOTE node
> [powerbook.2-net:00606] pls:rsh: executing: /usr/bin/ssh g4d003.3-net
> orted --debug --bootproxy 1 --name 0.0.2 --num_procs 6 --vpid_start 0
> --nodename g4d003.3-net --universe
> motte_at_powerbook.2-net:default-universe --nsreplica
> "0.0.0;tcp://192.168.2.3:49443" --gprreplica
> "0.0.0;tcp://192.168.2.3:49443" --mpi-call-yield 0
> [g4d003.3-net:00411] [0,0,2] setting up session dir with
> [g4d003.3-net:00411] universe default-universe
> [g4d003.3-net:00411] user motte
> [g4d003.3-net:00411] host g4d003.3-net
> [g4d003.3-net:00411] jobid 0
> [g4d003.3-net:00411] procid 2
> [g4d003.3-net:00411] procdir:
> /tmp/openmpi-sessions-motte_at_g4d003.3-net_0/default-universe/0/2
> [g4d003.3-net:00411] jobdir:
> /tmp/openmpi-sessions-motte_at_g4d003.3-net_0/default-universe/0
> [g4d003.3-net:00411] unidir:
> /tmp/openmpi-sessions-motte_at_g4d003.3-net_0/default-universe
> [g4d003.3-net:00411] top: openmpi-sessions-motte_at_g4d003.3-net_0
> [g4d003.3-net:00411] tmp: /tmp
> [powerbook.2-net:00606] pls:rsh: launching on node g4d002.3-net
> [powerbook.2-net:00606] pls:rsh: not oversubscribed -- setting
> mpi_yield_when_idle to 0
> [powerbook.2-net:00606] pls:rsh: g4d002.3-net is a REMOTE node
> [powerbook.2-net:00606] pls:rsh: executing: /usr/bin/ssh g4d002.3-net
> orted --debug --bootproxy 1 --name 0.0.3 --num_procs 6 --vpid_start 0
> --nodename g4d002.3-net --universe
> motte_at_powerbook.2-net:default-universe --nsreplica
> "0.0.0;tcp://192.168.2.3:49443" --gprreplica
> "0.0.0;tcp://192.168.2.3:49443" --mpi-call-yield 0
> [powerbook.2-net:00606] pls:rsh: launching on node g4d001.3-net
> [powerbook.2-net:00606] pls:rsh: not oversubscribed -- setting
> mpi_yield_when_idle to 0
> [powerbook.2-net:00606] pls:rsh: g4d001.3-net is a REMOTE node
> [powerbook.2-net:00606] pls:rsh: executing: /usr/bin/ssh g4d001.3-net
> orted --debug --bootproxy 1 --name 0.0.4 --num_procs 6 --vpid_start 0
> --nodename g4d001.3-net --universe
> motte_at_powerbook.2-net:default-universe --nsreplica
> "0.0.0;tcp://192.168.2.3:49443" --gprreplica
> "0.0.0;tcp://192.168.2.3:49443" --mpi-call-yield 0
> [powerbook.2-net:00606] pls:rsh: launching on node G5Dual.3-net
> [powerbook.2-net:00606] pls:rsh: not oversubscribed -- setting
> mpi_yield_when_idle to 0
> [powerbook.2-net:00606] pls:rsh: G5Dual.3-net is a REMOTE node
> [powerbook.2-net:00606] pls:rsh: executing: /usr/bin/ssh G5Dual.3-net
> orted --debug --bootproxy 1 --name 0.0.5 --num_procs 6 --vpid_start 0
> --nodename G5Dual.3-net --universe
> motte_at_powerbook.2-net:default-universe --nsreplica
> "0.0.0;tcp://192.168.2.3:49443" --gprreplica
> "0.0.0;tcp://192.168.2.3:49443" --mpi-call-yield 0
> [g4d001.3-net:00336] [0,0,4] setting up session dir with
> [g4d001.3-net:00336] universe default-universe
> [g4d001.3-net:00336] user motte
> [g4d001.3-net:00336] host g4d001.3-net
> [g4d001.3-net:00336] jobid 0
> [g4d001.3-net:00336] procid 4
> [g4d001.3-net:00336] procdir:
> /tmp/openmpi-sessions-motte_at_g4d001.3-net_0/default-universe/0/4
> [g4d001.3-net:00336] jobdir:
> /tmp/openmpi-sessions-motte_at_g4d001.3-net_0/default-universe/0
> [g4d001.3-net:00336] unidir:
> /tmp/openmpi-sessions-motte_at_g4d001.3-net_0/default-universe
> [g4d001.3-net:00336] top: openmpi-sessions-motte_at_g4d001.3-net_0
> [g4d001.3-net:00336] tmp: /tmp
> [g4d002.3-net:00279] [0,0,3] setting up session dir with
> [g4d002.3-net:00279] universe default-universe
> [g4d002.3-net:00279] user motte
> [g4d002.3-net:00279] host g4d002.3-net
> [g4d002.3-net:00279] jobid 0
> [g4d002.3-net:00279] procid 3
> [g4d002.3-net:00279] procdir:
> /tmp/openmpi-sessions-motte_at_g4d002.3-net_0/default-universe/0/3
> [g4d002.3-net:00279] jobdir:
> /tmp/openmpi-sessions-motte_at_g4d002.3-net_0/default-universe/0
> [g4d002.3-net:00279] unidir:
> /tmp/openmpi-sessions-motte_at_g4d002.3-net_0/default-universe
> [g4d002.3-net:00279] top: openmpi-sessions-motte_at_g4d002.3-net_0
> [g4d002.3-net:00279] tmp: /tmp
> [g5dual.3-net:00434] [0,0,5] setting up session dir with
> [g5dual.3-net:00434] universe default-universe
> [g5dual.3-net:00434] user motte
> [g5dual.3-net:00434] host G5Dual.3-net
> [g5dual.3-net:00434] jobid 0
> [g5dual.3-net:00434] procid 5
> [g5dual.3-net:00434] procdir:
> /tmp/openmpi-sessions-motte_at_G5Dual.3-net_0/default-universe/0/5
> [g5dual.3-net:00434] jobdir:
> /tmp/openmpi-sessions-motte_at_G5Dual.3-net_0/default-universe/0
> [g5dual.3-net:00434] unidir:
> /tmp/openmpi-sessions-motte_at_G5Dual.3-net_0/default-universe
> [g5dual.3-net:00434] top: openmpi-sessions-motte_at_G5Dual.3-net_0
> [g5dual.3-net:00434] tmp: /tmp
> [powerbook.2-net:00613] [0,1,4] setting up session dir with
> [powerbook.2-net:00613] universe default-universe
> [powerbook.2-net:00613] user motte
> [powerbook.2-net:00613] host Powerbook.2-net
> [powerbook.2-net:00613] jobid 1
> [powerbook.2-net:00613] procid 4
> [powerbook.2-net:00613] procdir:
> /tmp/openmpi-sessions-motte_at_Powerbook.2-net_0/default-universe/1/4
> [powerbook.2-net:00613] jobdir:
> /tmp/openmpi-sessions-motte_at_Powerbook.2-net_0/default-universe/1
> [powerbook.2-net:00613] unidir:
> /tmp/openmpi-sessions-motte_at_Powerbook.2-net_0/default-universe
> [powerbook.2-net:00613] top: openmpi-sessions-motte_at_Powerbook.2-net_0
> [powerbook.2-net:00613] tmp: /tmp
> [g5dual.3-net:00436] [0,1,0] setting up session dir with
> [g5dual.3-net:00436] universe default-universe
> [g5dual.3-net:00436] user motte
> [g5dual.3-net:00436] host G5Dual.3-net
> [g5dual.3-net:00436] jobid 1
> [g5dual.3-net:00436] procid 0
> [g5dual.3-net:00436] procdir:
> /tmp/openmpi-sessions-motte_at_G5Dual.3-net_0/default-universe/1/0
> [g5dual.3-net:00436] jobdir:
> /tmp/openmpi-sessions-motte_at_G5Dual.3-net_0/default-universe/1
> [g5dual.3-net:00436] unidir:
> /tmp/openmpi-sessions-motte_at_G5Dual.3-net_0/default-universe
> [g5dual.3-net:00436] top: openmpi-sessions-motte_at_G5Dual.3-net_0
> [g5dual.3-net:00436] tmp: /tmp
> [g4d001.3-net:00338] [0,1,1] setting up session dir with
> [g4d001.3-net:00338] universe default-universe
> [g4d001.3-net:00338] user motte
> [g4d001.3-net:00338] host g4d001.3-net
> [g4d001.3-net:00338] jobid 1
> [g4d001.3-net:00338] procid 1
> [g4d001.3-net:00338] procdir:
> /tmp/openmpi-sessions-motte_at_g4d001.3-net_0/default-universe/1/1
> [g4d001.3-net:00338] jobdir:
> /tmp/openmpi-sessions-motte_at_g4d001.3-net_0/default-universe/1
> [g4d001.3-net:00338] unidir:
> /tmp/openmpi-sessions-motte_at_g4d001.3-net_0/default-universe
> [g4d001.3-net:00338] top: openmpi-sessions-motte_at_g4d001.3-net_0
> [g4d001.3-net:00338] tmp: /tmp
> [g4d003.3-net:00413] [0,1,3] setting up session dir with
> [g4d003.3-net:00413] universe default-universe
> [g4d003.3-net:00413] user motte
> [g4d003.3-net:00413] host g4d003.3-net
> [g4d003.3-net:00413] jobid 1
> [g4d003.3-net:00413] procid 3
> [g4d003.3-net:00413] procdir:
> /tmp/openmpi-sessions-motte_at_g4d003.3-net_0/default-universe/1/3
> [g4d003.3-net:00413] jobdir:
> /tmp/openmpi-sessions-motte_at_g4d003.3-net_0/default-universe/1
> [g4d003.3-net:00413] unidir:
> /tmp/openmpi-sessions-motte_at_g4d003.3-net_0/default-universe
> [g4d003.3-net:00413] top: openmpi-sessions-motte_at_g4d003.3-net_0
> [g4d003.3-net:00413] tmp: /tmp
> [g4d002.3-net:00281] [0,1,2] setting up session dir with
> [g4d002.3-net:00281] universe default-universe
> [g4d002.3-net:00281] user motte
> [g4d002.3-net:00281] host g4d002.3-net
> [g4d002.3-net:00281] jobid 1
> [g4d002.3-net:00281] procid 2
> [g4d002.3-net:00281] procdir:
> /tmp/openmpi-sessions-motte_at_g4d002.3-net_0/default-universe/1/2
> [g4d002.3-net:00281] jobdir:
> /tmp/openmpi-sessions-motte_at_g4d002.3-net_0/default-universe/1
> [g4d002.3-net:00281] unidir:
> /tmp/openmpi-sessions-motte_at_g4d002.3-net_0/default-universe
> [g4d002.3-net:00281] top: openmpi-sessions-motte_at_g4d002.3-net_0
> [g4d002.3-net:00281] tmp: /tmp
> [powerbook.2-net:00606] spawn: in job_state_callback(jobid = 1, state = 0x4)
> [powerbook.2-net:00606] Info: Setting up debugger process table for
> applications
> MPIR_being_debugged = 0
> MPIR_debug_gate = 0
> MPIR_debug_state = 1
> MPIR_acquired_pre_main = 0
> MPIR_i_am_starter = 0
> MPIR_proctable_size = 5
> MPIR_proctable:
> (i, host, exe, pid) = (0, G5Dual.3-net,
> /Network/CFD/hello/hello_world, 436)
> (i, host, exe, pid) = (1, g4d001.3-net,
> /Network/CFD/hello/hello_world, 338)
> (i, host, exe, pid) = (2, g4d002.3-net,
> /Network/CFD/hello/hello_world, 281)
> (i, host, exe, pid) = (3, g4d003.3-net,
> /Network/CFD/hello/hello_world, 413)
> (i, host, exe, pid) = (4, Powerbook.2-net,
> /Network/CFD/hello/hello_world, 613)
> [powerbook.2-net:00613] [0,1,4] ompi_mpi_init completed
> [g4d001.3-net:00338] [0,1,1] ompi_mpi_init completed
> [g5dual.3-net:00436] [0,1,0] ompi_mpi_init completed
> [g4d003.3-net:00413] [0,1,3] ompi_mpi_init completed
> [g4d002.3-net:00281] [0,1,2] ompi_mpi_init completed
> node 1 :Hello, world
> node 2 :Hello, world node 3 :Hello, world
> [g5dual.3-net:00436] *** An error occurred in MPI_Send
>
> [g5dual.3-net:00436] *** on communicator MPI_COMM_WORLD
> [g5dual.3-net:00436] *** MPI_ERR_INTERN: internal error
> [g5dual.3-net:00436] *** MPI_ERRORS_ARE_FATAL (goodbye)
> --------------------------------------------------------------------------
> WARNING: A process refused to die!
>
> Host: powerbook.2-net
> PID: 613
>
> This process may still be running and/or consuming resources.
> --------------------------------------------------------------------------
> --------------------------------------------------------------------------
> WARNING: A process refused to die!
>
> Host: g4d003.3-net
> PID: 413
>
> This process may still be running and/or consuming resources.
> --------------------------------------------------------------------------
> --------------------------------------------------------------------------
> WARNING: A process refused to die!
>
> Host: g5dual.3-net
> PID: 436
>
> This process may still be running and/or consuming resources.
> --------------------------------------------------------------------------
> --------------------------------------------------------------------------
> WARNING: A process refused to die!
>
> Host: g4d002.3-net
> PID: 281
>
> This process may still be running and/or consuming resources.
> --------------------------------------------------------------------------
> --------------------------------------------------------------------------
> WARNING: A process refused to die!
>
> Host: g4d001.3-net
> PID: 338
>
> This process may still be running and/or consuming resources.
> --------------------------------------------------------------------------
> [g5dual.3-net:00434] sess_dir_finalize: found proc session dir empty -
> deleting
> [g5dual.3-net:00434] sess_dir_finalize: found job session dir empty -
> deleting
> [g5dual.3-net:00434] sess_dir_finalize: univ session dir not empty - leaving
> [powerbook.2-net:00607] orted: job_state_callback(jobid = 1, state =
> ORTE_PROC_STATE_ABORTED)
> [g5dual.3-net:00434] orted: job_state_callback(jobid = 1, state =
> ORTE_PROC_STATE_ABORTED)
> [g4d003.3-net:00411] orted: job_state_callback(jobid = 1, state =
> ORTE_PROC_STATE_ABORTED)
> [g4d001.3-net:00336] orted: job_state_callback(jobid = 1, state =
> ORTE_PROC_STATE_ABORTED)
> [g5dual.3-net:00434] sess_dir_finalize: job session dir not empty - leaving
> [g5dual.3-net:00434] sess_dir_finalize: found proc session dir empty -
> deleting
> [g5dual.3-net:00434] sess_dir_finalize: found job session dir empty -
> deleting
> [g5dual.3-net:00434] sess_dir_finalize: found univ session dir empty -
> deleting
> [g5dual.3-net:00434] sess_dir_finalize: found top session dir empty -
> deleting
> [g4d002.3-net:00279] orted: job_state_callback(jobid = 1, state =
> ORTE_PROC_STATE_ABORTED)
> [g4d002.3-net:00279] sess_dir_finalize: found job session dir empty -
> deleting
> [g4d002.3-net:00279] sess_dir_finalize: univ session dir not empty - leaving
> [g4d002.3-net:00279] sess_dir_finalize: proc session dir not empty - leaving
> --------------------------------------------------------------------------
> WARNING: A process refused to die!
>
> Host: g4d002.3-net
> PID: 281
>
> This process may still be running and/or consuming resources.
> --------------------------------------------------------------------------
> --------------------------------------------------------------------------
> WARNING: A process refused to die!
>
> Host: g4d002.3-net
> PID: 281
>
> This process may still be running and/or consuming resources.
> --------------------------------------------------------------------------
> [g4d002.3-net:00279] sess_dir_finalize: found proc session dir empty -
> deleting
> [g4d002.3-net:00279] sess_dir_finalize: found job session dir empty -
> deleting
> [g4d002.3-net:00279] sess_dir_finalize: found univ session dir empty -
> deleting
> [g4d002.3-net:00279] sess_dir_finalize: found top session dir empty -
> deleting
> [powerbook.2-net:00607] sess_dir_finalize: found job session dir empty -
> deleting
> [powerbook.2-net:00607] sess_dir_finalize: univ session dir not empty -
> leaving
> [powerbook.2-net:00607] sess_dir_finalize: proc session dir not empty -
> leaving
> --------------------------------------------------------------------------
> WARNING: A process refused to die!
>
> Host: powerbook.2-net
> PID: 613
>
> This process may still be running and/or consuming resources.
> --------------------------------------------------------------------------
> --------------------------------------------------------------------------
> WARNING: A process refused to die!
>
> Host: powerbook.2-net
> PID: 613
>
> This process may still be running and/or consuming resources.
> --------------------------------------------------------------------------
> [powerbook.2-net:00607] sess_dir_finalize: found proc session dir empty
> - deleting
> [powerbook.2-net:00607] sess_dir_finalize: job session dir not empty -
> leaving
> [g4d001.3-net:00336] sess_dir_finalize: found job session dir empty -
> deleting
> [g4d001.3-net:00336] sess_dir_finalize: univ session dir not empty - leaving
> [g4d001.3-net:00336] sess_dir_finalize: proc session dir not empty - leaving
> --------------------------------------------------------------------------
> WARNING: A process refused to die!
>
> Host: g4d001.3-net
> PID: 338
>
> This process may still be running and/or consuming resources.
> --------------------------------------------------------------------------
> --------------------------------------------------------------------------
> WARNING: A process refused to die!
>
> Host: g4d001.3-net
> PID: 338
>
> This process may still be running and/or consuming resources.
> --------------------------------------------------------------------------
> [g4d001.3-net:00336] sess_dir_finalize: found proc session dir empty -
> deleting
> [g4d001.3-net:00336] sess_dir_finalize: found job session dir empty -
> deleting
> [g4d001.3-net:00336] sess_dir_finalize: found univ session dir empty -
> deleting
> [g4d001.3-net:00336] sess_dir_finalize: found top session dir empty -
> deleting
> [g4d003.3-net:00411] sess_dir_finalize: found job session dir empty -
> deleting
> [g4d003.3-net:00411] sess_dir_finalize: univ session dir not empty - leaving
> [g4d003.3-net:00411] sess_dir_finalize: proc session dir not empty - leaving
> --------------------------------------------------------------------------
> WARNING: A process refused to die!
>
> Host: g4d003.3-net
> PID: 413
>
> This process may still be running and/or consuming resources.
> --------------------------------------------------------------------------
> --------------------------------------------------------------------------
> WARNING: A process refused to die!
>
> Host: g4d003.3-net
> PID: 413
>
> This process may still be running and/or consuming resources.
> --------------------------------------------------------------------------
> 1 process killed (possibly by Open MPI)
> [g4d003.3-net:00411] orted: job_state_callback(jobid = 1, state =
> ORTE_PROC_STATE_TERMINATED)
> [g4d003.3-net:00411] sess_dir_finalize: found proc session dir empty -
> deleting
> [g4d003.3-net:00411] sess_dir_finalize: found job session dir empty -
> deleting
> [g4d003.3-net:00411] sess_dir_finalize: found univ session dir empty -
> deleting
> [g4d003.3-net:00411] sess_dir_finalize: found top session dir empty -
> deleting
> [powerbook:/Network/CFD/hello] motte%
> _______________________________________________
> users mailing list
> users_at_[hidden]
> http://www.open-mpi.org/mailman/listinfo.cgi/users

-- 
Jeff Squyres
Server Virtualization Business Unit
Cisco Systems