Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: Re: [OMPI users] few Problems
From: Gus Correa (gus_at_[hidden])
Date: 2009-04-22 17:06:57


Hi Luis, list

To complement Jeff's recommendation,
see if this recipe to setup passwordless ssh connections helps.
If you use RSA keys instead of DSA, replace all "dsa" by "rsa":

http://www.sshkeychain.org/mirrors/SSH-with-Keys-HOWTO/SSH-with-Keys-HOWTO-4.html#ss4.3

I hope this helps.

Gus Correa
---------------------------------------------------------------------
Gustavo Correa
Lamont-Doherty Earth Observatory - Columbia University
Palisades, NY, 10964-8000 - USA
---------------------------------------------------------------------

Jeff Squyres wrote:
> It looks like you need to fix your password-less ssh problems first:
>
> > Permission denied, please try again.
> > AH72000_at_cluster-srv2's password:
>
> As I mentioned earlier, you need to be able to be able to run
>
> ssh cluster-srv2 uptime
>
> without being prompted for a password before Open MPI will work properly.
>
> If you're still having problems after fixing this, please send all the
> information from the "help" URL I sent earlier.
>
> Thanks!
>
>
> On Apr 22, 2009, at 3:24 PM, Luis Vitorio Cargnini wrote:
>
>> ok this is the debug information debug running on 5 nodes (trying at
>> least), the process is locked until now:
>>
>> each node is composed by two quad-core microprocessors.
>>
>> (don't finish), one node yet asked me the password. I have the home
>> partition mounted (the same) in all nodes. so login in cluster-
>> srv[0-4] is the same thing, I generated the

  key in each node, in
>> different files and added all to the authorized_keys, it should be
>> working.
>>
>> That is it, all help is welcome.
>>
>>
>> this is the code been executed:
>>
>> #include <stdio.h>
>> #include <mpi.h>
>>
>>
>> int main (int argc, char *argv[])
>> {
>> int rank, size;
>>
>> MPI_Init (&argc, &argv); /* starts MPI */
>> MPI_Comm_rank (MPI_COMM_WORLD, &rank); /* get current process
>> id */
>> MPI_Comm_size (MPI_COMM_WORLD, &size); /* get number of
>> processes */
>> printf( "Hello world from process %d of %d\n", rank, size );
>> MPI_Finalize();
>> return 0;
>> }
>>
>>
>> ------
>> debug:
>>
>> -bash-3.2$ mpirun -v -d -hostfile chosts -np 32 /export/cluster/home/
>> AH72000/mpi/hello
>>
>> [cluster-srv0:21606] procdir: /tmp/openmpi-sessions-AH72000_at_cluster-
>> srv0_0/35335/0/0
>> [cluster-srv0:21606] jobdir: /tmp/openmpi-sessions-AH72000_at_cluster-
>> srv0_0/35335/0
>> [cluster-srv0:21606] top: openmpi-sessions-AH72000_at_cluster-srv0_0
>> [cluster-srv0:21606] tmp: /tmp
>> [cluster-srv0:21606] mpirun: reset PATH: /export/cluster/appl/x86_64/
>> llvm/bin:/bin:/sbin:/export/cluster/appl/x86_64/llvm/bin:/usr/local/
>> llvm/bin:/usr/local/bin:/usr/bin:/usr/sbin:/home/GTI420/AH72000/oe/
>> bitbake/bin
>> [cluster-srv0:21606] mpirun: reset LD_LIBRARY_PATH: /export/cluster/
>> appl/x86_64/llvm/lib:/lib64:/lib:/export/cluster/appl/x86_64/llvm/lib:/
>> usr/lib64:/usr/lib:/usr/local/lib64:/usr/local/lib
>> AH72000_at_cluster-srv1's password: AH72000_at_cluster-srv2's password:
>> AH72000_at_cluster-srv3's password:
>> [cluster-srv1:07406] procdir: /tmp/openmpi-sessions-AH72000_at_cluster-
>> srv1_0/35335/0/1
>> [cluster-srv1:07406] jobdir: /tmp/openmpi-sessions-AH72000_at_cluster-
>> srv1_0/35335/0
>> [cluster-srv1:07406] top: openmpi-sessions-AH72000_at_cluster-srv1_0
>> [cluster-srv1:07406] tmp: /tmp
>>
>>
>> Permission denied, please try again.
>> AH72000_at_cluster-srv2's password:
>> [cluster-srv3:14230] procdir: /tmp/openmpi-sessions-AH72000_at_cluster-
>> srv3_0/35335/0/3
>> [cluster-srv3:14230] jobdir: /tmp/openmpi-sessions-AH72000_at_cluster-
>> srv3_0/35335/0
>> [cluster-srv3:14230] top: openmpi-sessions-AH72000_at_cluster-srv3_0
>> [cluster-srv3:14230] tmp: /tmp
>>
>> Permission denied, please try again.
>> AH72000_at_cluster-srv2's password:
>> [cluster-srv2:17092] procdir: /tmp/openmpi-sessions-AH72000_at_cluster-
>> srv2_0/35335/0/2
>> [cluster-srv2:17092] jobdir: /tmp/openmpi-sessions-AH72000_at_cluster-
>> srv2_0/35335/0
>> [cluster-srv2:17092] top: openmpi-sessions-AH72000_at_cluster-srv2_0
>> [cluster-srv2:17092] tmp: /tmp
>> [cluster-srv0:21606] [[35335,0],0] node[0].name cluster-srv0 daemon 0
>> arch ffc91200
>> [cluster-srv0:21606] [[35335,0],0] node[1].name cluster-srv1 daemon 1
>> arch ffc91200
>> [cluster-srv0:21606] [[35335,0],0] node[2].name cluster-srv2 daemon 2
>> arch ffc91200
>> [cluster-srv0:21606] [[35335,0],0] node[3].name cluster-srv3 daemon 3
>> arch ffc91200
>> [cluster-srv0:21606] [[35335,0],0] node[4].name cluster-srv4 daemon
>> INVALID arch ffc91200
>> [cluster-srv1:07406] [[35335,0],1] node[0].name cluster-srv0 daemon 0
>> arch ffc91200
>> [cluster-srv1:07406] [[35335,0],1] node[1].name cluster-srv1 daemon 1
>> arch ffc91200
>> [cluster-srv1:07406] [[35335,0],1] node[2].name cluster-srv2 daemon 2
>> arch ffc91200
>> [cluster-srv1:07406] [[35335,0],1] node[3].name cluster-srv3 daemon 3
>> arch ffc91200
>> [cluster-srv1:07406] [[35335,0],1] node[4].name cluster-srv4 daemon
>> INVALID arch ffc91200
>> [cluster-srv2:17092] [[35335,0],2] node[0].name cluster-srv0 daemon 0
>> arch ffc91200
>> [cluster-srv2:17092] [[35335,0],2] node[1].name cluster-srv1 daemon 1
>> arch ffc91200
>> [cluster-srv2:17092] [[35335,0],2] node[2].name cluster-srv2 daemon 2
>> arch ffc91200
>> [cluster-srv2:17092] [[35335,0],2] node[3].name cluster-srv3 daemon 3
>> arch ffc91200
>> [cluster-srv2:17092] [[35335,0],2] node[4].name cluster-srv4 daemon
>> INVALID arch ffc91200
>> [cluster-srv0:21611] procdir: /tmp/openmpi-sessions-AH72000_at_cluster-
>> srv0_0/35335/1/0
>> [cluster-srv0:21611] jobdir: /tmp/openmpi-sessions-AH72000_at_cluster-
>> srv0_0/35335/1
>> [cluster-srv0:21611] top: openmpi-sessions-AH72000_at_cluster-srv0_0
>> [cluster-srv0:21611] tmp: /tmp
>> [cluster-srv0:21613] procdir: /tmp/openmpi-sessions-AH72000_at_cluster-
>> srv0_0/35335/1/2
>> [cluster-srv0:21613] jobdir: /tmp/openmpi-sessions-AH72000_at_cluster-
>> srv0_0/35335/1
>> [cluster-srv0:21613] top: openmpi-sessions-AH72000_at_cluster-srv0_0
>> [cluster-srv0:21613] tmp: /tmp
>> [cluster-srv0:21612] procdir: /tmp/openmpi-sessions-AH72000_at_cluster-
>> srv0_0/35335/1/1
>> [cluster-srv0:21612] jobdir: /tmp/openmpi-sessions-AH72000_at_cluster-
>> srv0_0/35335/1
>> [cluster-srv0:21612] top: openmpi-sessions-AH72000_at_cluster-srv0_0
>> [cluster-srv0:21612] tmp: /tmp
>> [cluster-srv0:21614] procdir: /tmp/openmpi-sessions-AH72000_at_cluster-
>> srv0_0/35335/1/3
>> [cluster-srv0:21614] jobdir: /tmp/openmpi-sessions-AH72000_at_cluster-
>> srv0_0/35335/1
>> [cluster-srv0:21614] top: openmpi-sessions-AH72000_at_cluster-srv0_0
>> [cluster-srv0:21614] tmp: /tmp
>> [cluster-srv0:21615] procdir: /tmp/openmpi-sessions-AH72000_at_cluster-
>> srv0_0/35335/1/4
>> [cluster-srv0:21615] jobdir: /tmp/openmpi-sessions-AH72000_at_cluster-
>> srv0_0/35335/1
>> [cluster-srv0:21615] top: openmpi-sessions-AH72000_at_cluster-srv0_0
>> [cluster-srv0:21615] tmp: /tmp
>> [cluster-srv0:21616] procdir: /tmp/openmpi-sessions-AH72000_at_cluster-
>> srv0_0/35335/1/5
>> [cluster-srv0:21616] jobdir: /tmp/openmpi-sessions-AH72000_at_cluster-
>> srv0_0/35335/1
>> [cluster-srv0:21616] top: openmpi-sessions-AH72000_at_cluster-srv0_0
>> [cluster-srv0:21616] tmp: /tmp
>> [cluster-srv0:21617] procdir: /tmp/openmpi-sessions-AH72000_at_cluster-
>> srv0_0/35335/1/6
>> [cluster-srv0:21617] jobdir: /tmp/openmpi-sessions-AH72000_at_cluster-
>> srv0_0/35335/1
>> [cluster-srv0:21617] top: openmpi-sessions-AH72000_at_cluster-srv0_0
>> [cluster-srv0:21617] tmp: /tmp
>> [cluster-srv0:21618] procdir: /tmp/openmpi-sessions-AH72000_at_cluster-
>> srv0_0/35335/1/7
>> [cluster-srv0:21618] jobdir: /tmp/openmpi-sessions-AH72000_at_cluster-
>> srv0_0/35335/1
>> [cluster-srv0:21618] top: openmpi-sessions-AH72000_at_cluster-srv0_0
>> [cluster-srv0:21618] tmp: /tmp
>> [cluster-srv1:07408] procdir: /tmp/openmpi-sessions-AH72000_at_cluster-
>> srv1_0/35335/1/8
>> [cluster-srv1:07408] jobdir: /tmp/openmpi-sessions-AH72000_at_cluster-
>> srv1_0/35335/1
>> [cluster-srv1:07408] top: openmpi-sessions-AH72000_at_cluster-srv1_0
>> [cluster-srv1:07408] tmp: /tmp
>> [cluster-srv3:14230] [[35335,0],3] node[0].name cluster-srv0 daemon 0
>> arch ffc91200
>> [cluster-srv3:14230] [[35335,0],3] node[1].name cluster-srv1 daemon 1
>> arch ffc91200
>> [cluster-srv3:14230] [[35335,0],3] node[2].name cluster-srv2 daemon 2
>> arch ffc91200
>> [cluster-srv3:14230] [[35335,0],3] node[3].name cluster-srv3 daemon 3
>> arch ffc91200
>> [cluster-srv3:14230] [[35335,0],3] node[4].name cluster-srv4 daemon
>> INVALID arch ffc91200
>> [cluster-srv1:07409] procdir: /tmp/openmpi-sessions-AH72000_at_cluster-
>> srv1_0/35335/1/9
>> [cluster-srv1:07409] jobdir: /tmp/openmpi-sessions-AH72000_at_cluster-
>> srv1_0/35335/1
>> [cluster-srv1:07409] top: openmpi-sessions-AH72000_at_cluster-srv1_0
>> [cluster-srv1:07409] tmp: /tmp
>> [cluster-srv1:07410] procdir: /tmp/openmpi-sessions-AH72000_at_cluster-
>> srv1_0/35335/1/10
>> [cluster-srv1:07410] jobdir: /tmp/openmpi-sessions-AH72000_at_cluster-
>> srv1_0/35335/1
>> [cluster-srv1:07410] top: openmpi-sessions-AH72000_at_cluster-srv1_0
>> [cluster-srv1:07410] tmp: /tmp
>> [cluster-srv0:21606] Info: Setting up debugger process table for
>> applications
>> MPIR_being_debugged = 0
>> MPIR_debug_state = 1
>> MPIR_partial_attach_ok = 1
>> MPIR_i_am_starter = 0
>> MPIR_proctable_size = 32
>> MPIR_proctable:
>> (i, host, exe, pid) = (0, cluster-srv0, /export/cluster/home/
>> AH72000/mpi/hello, 21611)
>> (i, host, exe, pid) = (1, cluster-srv0, /export/cluster/home/
>> AH72000/mpi/hello, 21612)
>> (i, host, exe, pid) = (2, cluster-srv0, /export/cluster/home/
>> AH72000/mpi/hello, 21613)
>> (i, host, exe, pid) = (3, cluster-srv0, /export/cluster/home/
>> AH72000/mpi/hello, 21614)
>> (i, host, exe, pid) = (4, cluster-srv0, /export/cluster/home/
>> AH72000/mpi/hello, 21615)
>> (i, host, exe, pid) = (5, cluster-srv0, /export/cluster/home/
>> AH72000/mpi/hello, 21616)
>> (i, host, exe, pid) = (6, cluster-srv0, /export/cluster/home/
>> AH72000/mpi/hello, 21617)
>> (i, host, exe, pid) = (7, cluster-srv0, /export/cluster/home/
>> AH72000/mpi/hello, 21618)
>> (i, host, exe, pid) = (8, cluster-srv1, /export/cluster/home/
>> AH72000/mpi/hello, 7408)
>> (i, host, exe, pid) = (9, cluster-srv1, /export/cluster/home/
>> AH72000/mpi/hello, 7409)
>> (i, host, exe, pid) = (10, cluster-srv1, /export/cluster/home/
>> AH72000/mpi/hello, 7410)
>> (i, host, exe, pid) = (11, cluster-srv1, /export/cluster/home/
>> AH72000/mpi/hello, 7411)
>> (i, host, exe, pid) = (12, cluster-srv1, /export/cluster/home/
>> AH72000/mpi/hello, 7412)
>> (i, host, exe, pid) = (13, cluster-srv1, /export/cluster/home/
>> AH72000/mpi/hello, 7413)
>> (i, host, exe, pid) = (14, cluster-srv1, /export/cluster/home/
>> AH72000/mpi/hello, 7414)
>> (i, host, exe, pid) = (15, cluster-srv1, /export/cluster/home/
>> AH72000/mpi/hello, 7415)
>> (i, host, exe, pid) = (16, cluster-srv2, /export/cluster/home/
>> AH72000/mpi/hello, 17096)
>> (i, host, exe, pid) = (17, cluster-srv2, /export/cluster/home/
>> AH72000/mpi/hello, 17097)
>> (i, host, exe, pid) = (18, cluster-srv2, /export/cluster/home/
>> AH72000/mpi/hello, 17098)
>> (i, host, exe, pid) = (19, cluster-srv2, /export/cluster/home/
>> AH72000/mpi/hello, 17099)
>> (i, host, exe, pid) = (20, cluster-srv2, /export/cluster/home/
>> AH72000/mpi/hello, 17100)
>> (i, host, exe, pid) = (21, cluster-srv2, /export/cluster/home/
>> AH72000/mpi/hello, 17101)
>> (i, host, exe, pid) = (22, cluster-srv2, /export/cluster/home/
>> AH72000/mpi/hello, 17102)
>> (i, host, exe, pid) = (23, cluster-srv2, /export/cluster/home/
>> AH72000/mpi/hello, 17103)
>> (i, host, exe, pid) = (24, cluster-srv3, /export/cluster/home/
>> AH72000/mpi/hello, 14232)
>> (i, host, exe, pid) = (25, cluster-srv3, /export/cluster/home/
>> AH72000/mpi/hello, 14233)
>> (i, host, exe, pid) = (26, cluster-srv3, /export/cluster/home/
>> AH72000/mpi/hello, 14234)
>> (i, host, exe, pid) = (27, cluster-srv3, /export/cluster/home/
>> AH72000/mpi/hello, 14235)
>> (i, host, exe, pid) = (28, cluster-srv3, /export/cluster/home/
>> AH72000/mpi/hello, 14236)
>> (i, host, exe, pid) = (29, cluster-srv3, /export/cluster/home/
>> AH72000/mpi/hello, 14237)
>> (i, host, exe, pid) = (30, cluster-srv3, /export/cluster/home/
>> AH72000/mpi/hello, 14238)
>> (i, host, exe, pid) = (31, cluster-srv3, /export/cluster/home/
>> AH72000/mpi/hello, 14239)
>> [cluster-srv1:07412] procdir: /tmp/openmpi-sessions-AH72000_at_cluster-
>> srv1_0/35335/1/12
>> [cluster-srv1:07412] jobdir: /tmp/openmpi-sessions-AH72000_at_cluster-
>> srv1_0/35335/1
>> [cluster-srv1:07412] top: openmpi-sessions-AH72000_at_cluster-srv1_0
>> [cluster-srv1:07412] tmp: /tmp
>> [cluster-srv1:07411] procdir: /tmp/openmpi-sessions-AH72000_at_cluster-
>> srv1_0/35335/1/11
>> [cluster-srv1:07411] jobdir: /tmp/openmpi-sessions-AH72000_at_cluster-
>> srv1_0/35335/1
>> [cluster-srv1:07411] top: openmpi-sessions-AH72000_at_cluster-srv1_0
>> [cluster-srv1:07411] tmp: /tmp
>> [cluster-srv1:07414] procdir: /tmp/openmpi-sessions-AH72000_at_cluster-
>> srv1_0/35335/1/14
>> [cluster-srv1:07414] jobdir: /tmp/openmpi-sessions-AH72000_at_cluster-
>> srv1_0/35335/1
>> [cluster-srv1:07414] top: openmpi-sessions-AH72000_at_cluster-srv1_0
>> [cluster-srv1:07414] tmp: /tmp
>> [cluster-srv1:07415] procdir: /tmp/openmpi-sessions-AH72000_at_cluster-
>> srv1_0/35335/1/15
>> [cluster-srv1:07415] jobdir: /tmp/openmpi-sessions-AH72000_at_cluster-
>> srv1_0/35335/1
>> [cluster-srv1:07415] top: openmpi-sessions-AH72000_at_cluster-srv1_0
>> [cluster-srv1:07415] tmp: /tmp
>> [cluster-srv1:07413] procdir: /tmp/openmpi-sessions-AH72000_at_cluster-
>> srv1_0/35335/1/13
>> [cluster-srv1:07413] jobdir: /tmp/openmpi-sessions-AH72000_at_cluster-
>> srv1_0/35335/1
>> [cluster-srv1:07413] top: openmpi-sessions-AH72000_at_cluster-srv1_0
>> [cluster-srv1:07413] tmp: /tmp
>> [cluster-srv2:17096] procdir: /tmp/openmpi-sessions-AH72000_at_cluster-
>> srv2_0/35335/1/16
>> [cluster-srv2:17096] jobdir: /tmp/openmpi-sessions-AH72000_at_cluster-
>> srv2_0/35335/1
>> [cluster-srv2:17096] top: openmpi-sessions-AH72000_at_cluster-srv2_0
>> [cluster-srv2:17096] tmp: /tmp
>> [cluster-srv2:17097] procdir: /tmp/openmpi-sessions-AH72000_at_cluster-
>> srv2_0/35335/1/17
>> [cluster-srv2:17097] jobdir: /tmp/openmpi-sessions-AH72000_at_cluster-
>> srv2_0/35335/1
>> [cluster-srv2:17097] top: openmpi-sessions-AH72000_at_cluster-srv2_0
>> [cluster-srv2:17097] tmp: /tmp
>> [cluster-srv2:17099] procdir: /tmp/openmpi-sessions-AH72000_at_cluster-
>> srv2_0/35335/1/19
>> [cluster-srv2:17099] jobdir: /tmp/openmpi-sessions-AH72000_at_cluster-
>> srv2_0/35335/1
>> [cluster-srv2:17099] top: openmpi-sessions-AH72000_at_cluster-srv2_0
>> [cluster-srv2:17099] tmp: /tmp
>> [cluster-srv2:17098] procdir: /tmp/openmpi-sessions-AH72000_at_cluster-
>> srv2_0/35335/1/18
>> [cluster-srv2:17098] jobdir: /tmp/openmpi-sessions-AH72000_at_cluster-
>> srv2_0/35335/1
>> [cluster-srv2:17098] top: openmpi-sessions-AH72000_at_cluster-srv2_0
>> [cluster-srv2:17098] tmp: /tmp
>> [cluster-srv2:17100] procdir: /tmp/openmpi-sessions-AH72000_at_cluster-
>> srv2_0/35335/1/20
>> [cluster-srv2:17100] jobdir: /tmp/openmpi-sessions-AH72000_at_cluster-
>> srv2_0/35335/1
>> [cluster-srv2:17100] top: openmpi-sessions-AH72000_at_cluster-srv2_0
>> [cluster-srv2:17100] tmp: /tmp
>> [cluster-srv2:17102] procdir: /tmp/openmpi-sessions-AH72000_at_cluster-
>> srv2_0/35335/1/22
>> [cluster-srv2:17102] jobdir: /tmp/openmpi-sessions-AH72000_at_cluster-
>> srv2_0/35335/1
>> [cluster-srv2:17102] top: openmpi-sessions-AH72000_at_cluster-srv2_0
>> [cluster-srv2:17102] tmp: /tmp
>> [cluster-srv2:17103] procdir: /tmp/openmpi-sessions-AH72000_at_cluster-
>> srv2_0/35335/1/23
>> [cluster-srv2:17103] jobdir: /tmp/openmpi-sessions-AH72000_at_cluster-
>> srv2_0/35335/1
>> [cluster-srv2:17103] top: openmpi-sessions-AH72000_at_cluster-srv2_0
>> [cluster-srv2:17103] tmp: /tmp
>> [cluster-srv2:17101] procdir: /tmp/openmpi-sessions-AH72000_at_cluster-
>> srv2_0/35335/1/21
>> [cluster-srv2:17101] jobdir: /tmp/openmpi-sessions-AH72000_at_cluster-
>> srv2_0/35335/1
>> [cluster-srv2:17101] top: openmpi-sessions-AH72000_at_cluster-srv2_0
>> [cluster-srv2:17101] tmp: /tmp
>> [cluster-srv3:14232] procdir: /tmp/openmpi-sessions-AH72000_at_cluster-
>> srv3_0/35335/1/24
>> [cluster-srv3:14232] jobdir: /tmp/openmpi-sessions-AH72000_at_cluster-
>> srv3_0/35335/1
>> [cluster-srv3:14232] top: openmpi-sessions-AH72000_at_cluster-srv3_0
>> [cluster-srv3:14232] tmp: /tmp
>> [cluster-srv3:14235] procdir: /tmp/openmpi-sessions-AH72000_at_cluster-
>> srv3_0/35335/1/27
>> [cluster-srv3:14235] jobdir: /tmp/openmpi-sessions-AH72000_at_cluster-
>> srv3_0/35335/1
>> [cluster-srv3:14235] top: openmpi-sessions-AH72000_at_cluster-srv3_0
>> [cluster-srv3:14235] tmp: /tmp
>> [cluster-srv3:14233] procdir: /tmp/openmpi-sessions-AH72000_at_cluster-
>> srv3_0/35335/1/25
>> [cluster-srv3:14233] jobdir: /tmp/openmpi-sessions-AH72000_at_cluster-
>> srv3_0/35335/1
>> [cluster-srv3:14233] top: openmpi-sessions-AH72000_at_cluster-srv3_0
>> [cluster-srv3:14233] tmp: /tmp
>> [cluster-srv3:14234] procdir: /tmp/openmpi-sessions-AH72000_at_cluster-
>> srv3_0/35335/1/26
>> [cluster-srv3:14234] jobdir: /tmp/openmpi-sessions-AH72000_at_cluster-
>> srv3_0/35335/1
>> [cluster-srv3:14234] top: openmpi-sessions-AH72000_at_cluster-srv3_0
>> [cluster-srv3:14234] tmp: /tmp
>> [cluster-srv3:14236] procdir: /tmp/openmpi-sessions-AH72000_at_cluster-
>> srv3_0/35335/1/28
>> [cluster-srv3:14236] jobdir: /tmp/openmpi-sessions-AH72000_at_cluster-
>> srv3_0/35335/1
>> [cluster-srv3:14236] top: openmpi-sessions-AH72000_at_cluster-srv3_0
>> [cluster-srv3:14236] tmp: /tmp
>> [cluster-srv3:14237] procdir: /tmp/openmpi-sessions-AH72000_at_cluster-
>> srv3_0/35335/1/29
>> [cluster-srv3:14237] jobdir: /tmp/openmpi-sessions-AH72000_at_cluster-
>> srv3_0/35335/1
>> [cluster-srv3:14237] top: openmpi-sessions-AH72000_at_cluster-srv3_0
>> [cluster-srv3:14237] tmp: /tmp
>> [cluster-srv3:14238] procdir: /tmp/openmpi-sessions-AH72000_at_cluster-
>> srv3_0/35335/1/30
>> [cluster-srv3:14238] jobdir: /tmp/openmpi-sessions-AH72000_at_cluster-
>> srv3_0/35335/1
>> [cluster-srv3:14238] top: openmpi-sessions-AH72000_at_cluster-srv3_0
>> [cluster-srv3:14238] tmp: /tmp
>> [cluster-srv3:14239] procdir: /tmp/openmpi-sessions-AH72000_at_cluster-
>> srv3_0/35335/1/31
>> [cluster-srv3:14239] jobdir: /tmp/openmpi-sessions-AH72000_at_cluster-
>> srv3_0/35335/1
>> [cluster-srv3:14239] top: openmpi-sessions-AH72000_at_cluster-srv3_0
>> [cluster-srv3:14239] tmp: /tmp
>>
>> [cluster-srv0:21611] [[35335,1],0] node[0].name cluster-srv0 daemon 0
>> arch ffc91200
>> [cluster-srv0:21611] [[35335,1],0] node[1].name cluster-srv1 daemon 1
>> arch ffc91200
>> [cluster-srv0:21611] [[35335,1],0] node[2].name cluster-srv2 daemon 2
>> arch ffc91200
>> [cluster-srv0:21611] [[35335,1],0] node[3].name cluster-srv3 daemon 3
>> arch ffc91200
>> [cluster-srv0:21611] [[35335,1],0] node[4].name cluster-srv4 daemon
>> INVALID arch ffc91200
>> [cluster-srv0:21613] [[35335,1],2] node[0].name cluster-srv0 daemon 0
>> arch ffc91200
>> [cluster-srv0:21613] [[35335,1],2] node[1].name cluster-srv1 daemon 1
>> arch ffc91200
>> [cluster-srv0:21613] [[35335,1],2] node[2].name cluster-srv2 daemon 2
>> arch ffc91200
>> [cluster-srv0:21613] [[35335,1],2] node[3].name cluster-srv3 daemon 3
>> arch ffc91200
>> [cluster-srv0:21613] [[35335,1],2] node[4].name cluster-srv4 daemon
>> INVALID arch ffc91200
>> [cluster-srv0:21614] [[35335,1],3] node[0].name cluster-srv0 daemon 0
>> arch ffc91200
>> [cluster-srv0:21614] [[35335,1],3] node[1].name cluster-srv1 daemon 1
>> arch ffc91200
>> [cluster-srv0:21614] [[35335,1],3] node[2].name cluster-srv2 daemon 2
>> arch ffc91200
>> [cluster-srv0:21614] [[35335,1],3] node[3].name cluster-srv3 daemon 3
>> arch ffc91200
>> [cluster-srv0:21614] [[35335,1],3] node[4].name cluster-srv4 daemon
>> INVALID arch ffc91200
>> [cluster-srv0:21612] [[35335,1],1] node[0].name cluster-srv0 daemon 0
>> arch ffc91200
>> [cluster-srv0:21612] [[35335,1],1] node[1].name cluster-srv1 daemon 1
>> arch ffc91200
>> [cluster-srv0:21612] [[35335,1],1] node[2].name cluster-srv2 daemon 2
>> arch ffc91200
>> [cluster-srv0:21612] [[35335,1],1] node[3].name cluster-srv3 daemon 3
>> arch ffc91200
>> [cluster-srv0:21612] [[35335,1],1] node[4].name cluster-srv4 daemon
>> INVALID arch ffc91200
>> [cluster-srv0:21615] [[35335,1],4] node[0].name cluster-srv0 daemon 0
>> arch ffc91200
>> [cluster-srv0:21615] [[35335,1],4] node[1].name cluster-srv1 daemon 1
>> arch ffc91200
>> [cluster-srv0:21615] [[35335,1],4] node[2].name cluster-srv2 daemon 2
>> arch ffc91200
>> [cluster-srv0:21615] [[35335,1],4] node[3].name cluster-srv3 daemon 3
>> arch ffc91200
>> [cluster-srv0:21615] [[35335,1],4] node[4].name cluster-srv4 daemon
>> INVALID arch ffc91200
>> [cluster-srv1:07408] [[35335,1],8] node[0].name cluster-srv0 daemon 0
>> arch ffc91200
>> [cluster-srv1:07408] [[35335,1],8] node[1].name cluster-srv1 daemon 1
>> arch ffc91200
>> [cluster-srv1:07408] [[35335,1],8] node[2].name cluster-srv2 daemon 2
>> arch ffc91200
>> [cluster-srv1:07408] [[35335,1],8] node[3].name cluster-srv3 daemon 3
>> arch ffc91200
>> [cluster-srv1:07408] [[35335,1],8] node[4].name cluster-srv4 daemon
>> INVALID arch ffc91200
>> [cluster-srv1:07409] [[35335,1],9] node[0].name cluster-srv0 daemon 0
>> arch ffc91200
>> [cluster-srv1:07409] [[35335,1],9] node[1].name cluster-srv1 daemon 1
>> arch ffc91200
>> [cluster-srv1:07409] [[35335,1],9] node[2].name cluster-srv2 daemon 2
>> arch ffc91200
>> [cluster-srv1:07409] [[35335,1],9] node[3].name cluster-srv3 daemon 3
>> arch ffc91200
>> [cluster-srv1:07409] [[35335,1],9] node[4].name cluster-srv4 daemon
>> INVALID arch ffc91200
>> [cluster-srv1:07410] [[35335,1],10] node[0].name cluster-srv0 daemon 0
>> arch ffc91200
>> [cluster-srv1:07410] [[35335,1],10] node[1].name cluster-srv1 daemon 1
>> arch ffc91200
>> [cluster-srv1:07410] [[35335,1],10] node[2].name cluster-srv2 daemon 2
>> arch ffc91200
>> [cluster-srv1:07410] [[35335,1],10] node[3].name cluster-srv3 daemon 3
>> arch ffc91200
>> [cluster-srv1:07410] [[35335,1],10] node[4].name cluster-srv4 daemon
>> INVALID arch ffc91200
>> [cluster-srv2:17096] [[35335,1],16] node[0].name cluster-srv0 daemon 0
>> arch ffc91200
>> [cluster-srv2:17096] [[35335,1],16] node[1].name cluster-srv1 daemon 1
>> arch ffc91200
>> [cluster-srv2:17096] [[35335,1],16] node[2].name cluster-srv2 daemon 2
>> arch ffc91200
>> [cluster-srv2:17096] [[35335,1],16] node[3].name cluster-srv3 daemon 3
>> arch ffc91200
>> [cluster-srv2:17096] [[35335,1],16] node[4].name cluster-srv4 daemon
>> INVALID arch ffc91200
>> [cluster-srv2:17097] [[35335,1],17] node[0].name cluster-srv0 daemon 0
>> arch ffc91200
>> [cluster-srv2:17097] [[35335,1],17] node[1].name cluster-srv1 daemon 1
>> arch ffc91200
>> [cluster-srv2:17097] [[35335,1],17] node[2].name cluster-srv2 daemon 2
>> arch ffc91200
>> [cluster-srv2:17097] [[35335,1],17] node[3].name cluster-srv3 daemon 3
>> arch ffc91200
>> [cluster-srv2:17097] [[35335,1],17] node[4].name cluster-srv4 daemon
>> INVALID arch ffc91200
>> [cluster-srv0:21617] [[35335,1],6] node[0].name cluster-srv0 daemon 0
>> arch ffc91200
>> [cluster-srv0:21617] [[35335,1],6] node[1].name cluster-srv1 daemon 1
>> arch ffc91200
>> [cluster-srv0:21617] [[35335,1],6] node[2].name cluster-srv2 daemon 2
>> arch ffc91200
>> [cluster-srv0:21617] [[35335,1],6] node[3].name cluster-srv3 daemon 3
>> arch ffc91200
>> [cluster-srv0:21617] [[35335,1],6] node[4].name cluster-srv4 daemon
>> INVALID arch ffc91200
>> [cluster-srv0:21616] [[35335,1],5] node[0].name cluster-srv0 daemon 0
>> arch ffc91200
>> [cluster-srv0:21616] [[35335,1],5] node[1].name cluster-srv1 daemon 1
>> arch ffc91200
>> [cluster-srv0:21616] [[35335,1],5] node[2].name cluster-srv2 daemon 2
>> arch ffc91200
>> [cluster-srv0:21616] [[35335,1],5] node[3].name cluster-srv3 daemon 3
>> arch ffc91200
>> [cluster-srv0:21616] [[35335,1],5] node[4].name cluster-srv4 daemon
>> INVALID arch ffc91200
>> [cluster-srv1:07411] [[35335,1],11] node[0].name cluster-srv0 daemon 0
>> arch ffc91200
>> [cluster-srv1:07411] [[35335,1],11] node[1].name cluster-srv1 daemon 1
>> arch ffc91200
>> [cluster-srv1:07411] [[35335,1],11] node[2].name cluster-srv2 daemon 2
>> arch ffc91200
>> [cluster-srv1:07411] [[35335,1],11] node[3].name cluster-srv3 daemon 3
>> arch ffc91200
>> [cluster-srv1:07411] [[35335,1],11] node[4].name cluster-srv4 daemon
>> INVALID arch ffc91200
>> [cluster-srv2:17099] [[35335,1],19] node[0].name cluster-srv0 daemon 0
>> arch ffc91200
>> [cluster-srv2:17099] [[35335,1],19] node[1].name cluster-srv1 daemon 1
>> arch ffc91200
>> [cluster-srv2:17099] [[35335,1],19] node[2].name cluster-srv2 daemon 2
>> arch ffc91200
>> [cluster-srv2:17099] [[35335,1],19] node[3].name cluster-srv3 daemon 3
>> arch ffc91200
>> [cluster-srv2:17099] [[35335,1],19] node[4].name cluster-srv4 daemon
>> INVALID arch ffc91200
>> [cluster-srv0:21618] [[35335,1],7] node[0].name cluster-srv0 daemon 0
>> arch ffc91200
>> [cluster-srv0:21618] [[35335,1],7] node[1].name cluster-srv1 daemon 1
>> arch ffc91200
>> [cluster-srv0:21618] [[35335,1],7] node[2].name cluster-srv2 daemon 2
>> arch ffc91200
>> [cluster-srv0:21618] [[35335,1],7] node[3].name cluster-srv3 daemon 3
>> arch ffc91200
>> [cluster-srv0:21618] [[35335,1],7] node[4].name cluster-srv4 daemon
>> INVALID arch ffc91200
>> [cluster-srv2:17098] [[35335,1],18] node[0].name cluster-srv0 daemon 0
>> arch ffc91200
>> [cluster-srv2:17098] [[35335,1],18] node[1].name cluster-srv1 daemon 1
>> arch ffc91200
>> [cluster-srv2:17098] [[35335,1],18] node[2].name cluster-srv2 daemon 2
>> arch ffc91200
>> [cluster-srv2:17098] [[35335,1],18] node[3].name cluster-srv3 daemon 3
>> arch ffc91200
>> [cluster-srv2:17098] [[35335,1],18] node[4].name cluster-srv4 daemon
>> INVALID arch ffc91200
>> [cluster-srv1:07412] [[35335,1],12] node[0].name cluster-srv0 daemon 0
>> arch ffc91200
>> [cluster-srv1:07412] [[35335,1],12] node[1].name cluster-srv1 daemon 1
>> arch ffc91200
>> [cluster-srv1:07412] [[35335,1],12] node[2].name cluster-srv2 daemon 2
>> arch ffc91200
>> [cluster-srv1:07412] [[35335,1],12] node[3].name cluster-srv3 daemon 3
>> arch ffc91200
>> [cluster-srv1:07412] [[35335,1],12] node[4].name cluster-srv4 daemon
>> INVALID arch ffc91200
>> [cluster-srv2:17102] [[35335,1],22] node[0].name cluster-srv0 daemon 0
>> arch ffc91200
>> [cluster-srv2:17102] [[35335,1],22] node[1].name cluster-srv1 daemon 1
>> arch ffc91200
>> [cluster-srv2:17102] [[35335,1],22] node[2].name cluster-srv2 daemon 2
>> arch ffc91200
>> [cluster-srv2:17102] [[35335,1],22] node[3].name cluster-srv3 daemon 3
>> arch ffc91200
>> [cluster-srv2:17102] [[35335,1],22] node[4].name cluster-srv4 daemon
>> INVALID arch ffc91200
>> [cluster-srv2:17100] [[35335,1],20] node[0].name cluster-srv0 daemon 0
>> arch ffc91200
>> [cluster-srv2:17100] [[35335,1],20] node[1].name cluster-srv1 daemon 1
>> arch ffc91200
>> [cluster-srv2:17100] [[35335,1],20] node[2].name cluster-srv2 daemon 2
>> arch ffc91200
>> [cluster-srv2:17100] [[35335,1],20] node[3].name cluster-srv3 daemon 3
>> arch ffc91200
>> [cluster-srv2:17100] [[35335,1],20] node[4].name cluster-srv4 daemon
>> INVALID arch ffc91200
>> [cluster-srv1:07415] [[35335,1],15] node[0].name cluster-srv0 daemon 0
>> arch ffc91200
>> [cluster-srv1:07415] [[35335,1],15] node[1].name cluster-srv1 daemon 1
>> arch ffc91200
>> [cluster-srv1:07415] [[35335,1],15] node[2].name cluster-srv2 daemon 2
>> arch ffc91200
>> [cluster-srv1:07415] [[35335,1],15] node[3].name cluster-srv3 daemon 3
>> arch ffc91200
>> [cluster-srv1:07415] [[35335,1],15] node[4].name cluster-srv4 daemon
>> INVALID arch ffc91200
>> [cluster-srv1:07414] [[35335,1],14] node[0].name cluster-srv0 daemon 0
>> arch ffc91200
>> [cluster-srv1:07414] [[35335,1],14] node[1].name cluster-srv1 daemon 1
>> arch ffc91200
>> [cluster-srv1:07414] [[35335,1],14] node[2].name cluster-srv2 daemon 2
>> arch ffc91200
>> [cluster-srv1:07414] [[35335,1],14] node[3].name cluster-srv3 daemon 3
>> arch ffc91200
>> [cluster-srv1:07414] [[35335,1],14] node[4].name cluster-srv4 daemon
>> INVALID arch ffc91200
>> [cluster-srv2:17101] [[35335,1],21] node[0].name cluster-srv0 daemon 0
>> arch ffc91200
>> [cluster-srv2:17101] [[35335,1],21] node[1].name cluster-srv1 daemon 1
>> arch ffc91200
>> [cluster-srv2:17101] [[35335,1],21] node[2].name cluster-srv2 daemon 2
>> arch ffc91200
>> [cluster-srv2:17101] [[35335,1],21] node[3].name cluster-srv3 daemon 3
>> arch ffc91200
>> [cluster-srv2:17101] [[35335,1],21] node[4].name cluster-srv4 daemon
>> INVALID arch ffc91200
>> [cluster-srv2:17103] [[35335,1],23] node[0].name cluster-srv0 daemon 0
>> arch ffc91200
>> [cluster-srv2:17103] [[35335,1],23] node[1].name cluster-srv1 daemon 1
>> arch ffc91200
>> [cluster-srv2:17103] [[35335,1],23] node[2].name cluster-srv2 daemon 2
>> arch ffc91200
>> [cluster-srv2:17103] [[35335,1],23] node[3].name cluster-srv3 daemon 3
>> arch ffc91200
>> [cluster-srv2:17103] [[35335,1],23] node[4].name cluster-srv4 daemon
>> INVALID arch ffc91200
>> [cluster-srv1:07413] [[35335,1],13] node[0].name cluster-srv0 daemon 0
>> arch ffc91200
>> [cluster-srv1:07413] [[35335,1],13] node[1].name cluster-srv1 daemon 1
>> arch ffc91200
>> [cluster-srv1:07413] [[35335,1],13] node[2].name cluster-srv2 daemon 2
>> arch ffc91200
>> [cluster-srv1:07413] [[35335,1],13] node[3].name cluster-srv3 daemon 3
>> arch ffc91200
>> [cluster-srv1:07413] [[35335,1],13] node[4].name cluster-srv4 daemon
>> INVALID arch ffc91200
>> [cluster-srv3:14232] [[35335,1],24] node[0].name cluster-srv0 daemon 0
>> arch ffc91200
>> [cluster-srv3:14232] [[35335,1],24] node[1].name cluster-srv1 daemon 1
>> arch ffc91200
>> [cluster-srv3:14232] [[35335,1],24] node[2].name cluster-srv2 daemon 2
>> arch ffc91200
>> [cluster-srv3:14232] [[35335,1],24] node[3].name cluster-srv3 daemon 3
>> arch ffc91200
>> [cluster-srv3:14232] [[35335,1],24] node[4].name cluster-srv4 daemon
>> INVALID arch ffc91200
>> [cluster-srv3:14235] [[35335,1],27] node[0].name cluster-srv0 daemon 0
>> arch ffc91200
>> [cluster-srv3:14235] [[35335,1],27] node[1].name cluster-srv1 daemon 1
>> arch ffc91200
>> [cluster-srv3:14235] [[35335,1],27] node[2].name cluster-srv2 daemon 2
>> arch ffc91200
>> [cluster-srv3:14235] [[35335,1],27] node[3].name cluster-srv3 daemon 3
>> arch ffc91200
>> [cluster-srv3:14235] [[35335,1],27] node[4].name cluster-srv4 daemon
>> INVALID arch ffc91200
>> [cluster-srv3:14233] [[35335,1],25] node[0].name cluster-srv0 daemon 0
>> arch ffc91200
>> [cluster-srv3:14233] [[35335,1],25] node[1].name cluster-srv1 daemon 1
>> arch ffc91200
>> [cluster-srv3:14233] [[35335,1],25] node[2].name cluster-srv2 daemon 2
>> arch ffc91200
>> [cluster-srv3:14233] [[35335,1],25] node[3].name cluster-srv3 daemon 3
>> arch ffc91200
>> [cluster-srv3:14233] [[35335,1],25] node[4].name cluster-srv4 daemon
>> INVALID arch ffc91200
>> [cluster-srv3:14234] [[35335,1],26] node[0].name cluster-srv0 daemon 0
>> arch ffc91200
>> [cluster-srv3:14234] [[35335,1],26] node[1].name cluster-srv1 daemon 1
>> arch ffc91200
>> [cluster-srv3:14234] [[35335,1],26] node[2].name cluster-srv2 daemon 2
>> arch ffc91200
>> [cluster-srv3:14234] [[35335,1],26] node[3].name cluster-srv3 daemon 3
>> arch ffc91200
>> [cluster-srv3:14234] [[35335,1],26] node[4].name cluster-srv4 daemon
>> INVALID arch ffc91200
>> [cluster-srv3:14236] [[35335,1],28] node[0].name cluster-srv0 daemon 0
>> arch ffc91200
>> [cluster-srv3:14236] [[35335,1],28] node[1].name cluster-srv1 daemon 1
>> arch ffc91200
>> [cluster-srv3:14236] [[35335,1],28] node[2].name cluster-srv2 daemon 2
>> arch ffc91200
>> [cluster-srv3:14236] [[35335,1],28] node[3].name cluster-srv3 daemon 3
>> arch ffc91200
>> [cluster-srv3:14236] [[35335,1],28] node[4].name cluster-srv4 daemon
>> INVALID arch ffc91200
>> [cluster-srv3:14237] [[35335,1],29] node[0].name cluster-srv0 daemon 0
>> arch ffc91200
>> [cluster-srv3:14237] [[35335,1],29] node[1].name cluster-srv1 daemon 1
>> arch ffc91200
>> [cluster-srv3:14237] [[35335,1],29] node[2].name cluster-srv2 daemon 2
>> arch ffc91200
>> [cluster-srv3:14237] [[35335,1],29] node[3].name cluster-srv3 daemon 3
>> arch ffc91200
>> [cluster-srv3:14237] [[35335,1],29] node[4].name cluster-srv4 daemon
>> INVALID arch ffc91200
>> [cluster-srv3:14239] [[35335,1],31] node[0].name cluster-srv0 daemon 0
>> arch ffc91200
>> [cluster-srv3:14239] [[35335,1],31] node[1].name cluster-srv1 daemon 1
>> arch ffc91200
>> [cluster-srv3:14239] [[35335,1],31] node[2].name cluster-srv2 daemon 2
>> arch ffc91200
>> [cluster-srv3:14239] [[35335,1],31] node[3].name cluster-srv3 daemon 3
>> arch ffc91200
>> [cluster-srv3:14239] [[35335,1],31] node[4].name cluster-srv4 daemon
>> INVALID arch ffc91200
>> [cluster-srv3:14238] [[35335,1],30] node[0].name cluster-srv0 daemon 0
>> arch ffc91200
>> [cluster-srv3:14238] [[35335,1],30] node[1].name cluster-srv1 daemon 1
>> arch ffc91200
>> [cluster-srv3:14238] [[35335,1],30] node[2].name cluster-srv2 daemon 2
>> arch ffc91200
>> [cluster-srv3:14238] [[35335,1],30] node[3].name cluster-srv3 daemon 3
>> arch ffc91200
>> [cluster-srv3:14238] [[35335,1],30] node[4].name cluster-srv4 daemon
>> INVALID arch ffc91200
>> Hello world from process 0 of 32
>> Hello world from process 4 of 32
>> Hello world from process 1 of 32
>> Hello world from process 7 of 32
>> Hello world from process 6 of 32
>> Hello world from process 24 of 32
>> Hello world from process 5 of 32
>> Hello world from process 27 of 32
>> Hello world from process 25 of 32
>> Hello world from process 21 of 32
>> Hello world from process 20 of 32
>> Hello world from process 18 of 32
>> Hello world from process 10 of 32
>> Hello world from process 16 of 32
>> Hello world from process 8 of 32
>> Hello world from process 12 of 32
>> Hello world from process 22 of 32
>> Hello world from process 13 of 32
>> Hello world from process 11 of 32
>> Hello world from process 14 of 32
>> Hello world from process 31 of 32
>> Hello world from process 19 of 32
>> Hello world from process 17 of 32
>> Hello world from process 26 of 32
>> Hello world from process 29 of 32
>> Hello world from process 9 of 32
>> Hello world from process 30 of 32
>> Hello world from process 2 of 32
>> Hello world from process 23 of 32
>> Hello world from process 3 of 32
>> Hello world from process 15 of 32
>> Hello world from process 28 of 32
>> [cluster-srv0:21615] sess_dir_finalize: proc session dir not empty -
>> leaving
>> [cluster-srv1:07409] sess_dir_finalize: proc session dir not empty -
>> leaving
>> [cluster-srv2:17099] sess_dir_finalize: proc session dir not empty -
>> leaving
>> [cluster-srv0:21606] sess_dir_finalize: proc session dir not empty -
>> leaving
>> [cluster-srv3:14233] sess_dir_finalize: proc session dir not empty -
>> leaving
>> [cluster-srv2:17098] sess_dir_finalize: proc session dir not empty -
>> leaving
>> [cluster-srv2:17092] sess_dir_finalize: proc session dir not empty -
>> leaving
>> [cluster-srv3:14230] sess_dir_finalize: proc session dir not empty -
>> leaving
>> [cluster-srv2:17092] sess_dir_finalize: proc session dir not empty -
>> leaving
>> [cluster-srv2:17097] sess_dir_finalize: proc session dir not empty -
>> leaving
>> [cluster-srv2:17092] sess_dir_finalize: proc session dir not empty -
>> leaving
>> [cluster-srv2:17096] sess_dir_finalize: proc session dir not empty -
>> leaving
>> [cluster-srv2:17092] sess_dir_finalize: proc session dir not empty -
>> leaving
>> [cluster-srv0:21617] sess_dir_finalize: proc session dir not empty -
>> leaving
>> [cluster-srv1:07415] sess_dir_finalize: proc session dir not empty -
>> leaving
>> [cluster-srv1:07414] sess_dir_finalize: proc session dir not empty -
>> leaving
>> [cluster-srv0:21606] sess_dir_finalize: proc session dir not empty -
>> leaving
>> [cluster-srv1:07406] sess_dir_finalize: proc session dir not empty -
>> leaving
>> [cluster-srv1:07406] sess_dir_finalize: proc session dir not empty -
>> leaving
>> [cluster-srv1:07413] sess_dir_finalize: proc session dir not empty -
>> leaving
>> [cluster-srv3:14237] sess_dir_finalize: proc session dir not empty -
>> leaving
>> [cluster-srv1:07406] sess_dir_finalize: proc session dir not empty -
>> leaving
>> [cluster-srv1:07406] sess_dir_finalize: proc session dir not empty -
>> leaving
>> [cluster-srv0:21613] sess_dir_finalize: proc session dir not empty -
>> leaving
>> [cluster-srv3:14230] sess_dir_finalize: proc session dir not empty -
>> leaving
>> [cluster-srv2:17101] sess_dir_finalize: proc session dir not empty -
>> leaving
>> [cluster-srv2:17102] sess_dir_finalize: proc session dir not empty -
>> leaving
>> [cluster-srv2:17100] sess_dir_finalize: proc session dir not empty -
>> leaving
>> [cluster-srv0:21606] sess_dir_finalize: proc session dir not empty -
>> leaving
>> [cluster-srv3:14234] sess_dir_finalize: proc session dir not empty -
>> leaving
>> [cluster-srv3:14235] sess_dir_finalize: proc session dir not empty -
>> leaving
>> [cluster-srv3:14230] sess_dir_finalize: proc session dir not empty -
>> leaving
>> [cluster-srv2:17092] sess_dir_finalize: proc session dir not empty -
>> leaving
>> [cluster-srv3:14230] sess_dir_finalize: proc session dir not empty -
>> leaving
>> [cluster-srv2:17103] sess_dir_finalize: proc session dir not empty -
>> leaving
>> [cluster-srv2:17092] sess_dir_finalize: proc session dir not empty -
>> leaving
>> [cluster-srv2:17092] sess_dir_finalize: proc session dir not empty -
>> leaving
>> [cluster-srv2:17092] sess_dir_finalize: proc session dir not empty -
>> leaving
>> [cluster-srv1:07412] sess_dir_finalize: proc session dir not empty -
>> leaving
>> [cluster-srv1:07408] sess_dir_finalize: proc session dir not empty -
>> leaving
>> [cluster-srv1:07411] sess_dir_finalize: proc session dir not empty -
>> leaving
>> [cluster-srv3:14238] sess_dir_finalize: proc session dir not empty -
>> leaving
>> [cluster-srv0:21616] sess_dir_finalize: proc session dir not empty -
>> leaving
>> [cluster-srv0:21614] sess_dir_finalize: proc session dir not empty -
>> leaving
>> [cluster-srv3:14236] sess_dir_finalize: proc session dir not empty -
>> leaving
>> [cluster-srv1:07406] sess_dir_finalize: proc session dir not empty -
>> leaving
>> [cluster-srv1:07406] sess_dir_finalize: proc session dir not empty -
>> leaving
>> [cluster-srv3:14239] sess_dir_finalize: proc session dir not empty -
>> leaving
>> [cluster-srv0:21606] sess_dir_finalize: proc session dir not empty -
>> leaving
>> [cluster-srv0:21606] sess_dir_finalize: proc session dir not empty -
>> leaving
>> [cluster-srv3:14232] sess_dir_finalize: proc session dir not empty -
>> leaving
>> [cluster-srv3:14230] sess_dir_finalize: proc session dir not empty -
>> leaving
>> [cluster-srv1:07406] sess_dir_finalize: proc session dir not empty -
>> leaving
>> [cluster-srv3:14230] sess_dir_finalize: proc session dir not empty -
>> leaving
>> [cluster-srv3:14230] sess_dir_finalize: proc session dir not empty -
>> leaving
>> [cluster-srv1:07410] sess_dir_finalize: proc session dir not empty -
>> leaving
>> [cluster-srv3:14230] sess_dir_finalize: proc session dir not empty -
>> leaving
>> [cluster-srv0:21612] sess_dir_finalize: proc session dir not empty -
>> leaving
>> [cluster-srv1:07406] sess_dir_finalize: proc session dir not empty -
>> leaving
>> [cluster-srv0:21611] sess_dir_finalize: proc session dir not empty -
>> leaving
>> [cluster-srv0:21606] sess_dir_finalize: proc session dir not empty -
>> leaving
>> [cluster-srv0:21606] sess_dir_finalize: proc session dir not empty -
>> leaving
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> Le 09-04-22 à 14:10, Jeff Squyres a écrit :
>>
>> > This isn't really enough information for us to help you. Can you
>> > send all the information here:
>> >
>> > http://www.open-mpi.org/community/help/
>> >
>> > Thanks.
>> >
>> >
>> > On Apr 21, 2009, at 10:34 AM, Luis Vitorio Cargnini wrote:
>> >
>> >> Hi,
>> >> Please someone can answer me which can be this problem ?
>> >> daemon INVALID arch ffc91200
>> >>
>> >>
>> >>
>> >>
>> >> the debug output:
>> >> [[41704,1],14] node[4].name cluster-srv4 daemon INVALID arch ffc91200
>> >> [cluster-srv3:09684] [[41704,1],13] node[0].name cluster-srv0
>> >> daemon 0
>> >> arch ffc91200
>> >> [cluster-srv3:09684] [[41704,1],13] node[1].name cluster-srv1
>> >> daemon 1
>> >> arch ffc91200
>> >> [cluster-srv3:09684] [[41704,1],13] node[2].name cluster-srv2
>> >> daemon 2
>> >> arch ffc91200
>> >> [cluster-srv3:09684] [[41704,1],13] node[3].name cluster-srv3
>> >> daemon 3
>> >> arch ffc91200
>> >> [cluster-srv3:09684] [[41704,1],13] node[4].name cluster-srv4 daemon
>> >> INVALID arch ffc91200
>> >>
>> >> ORTE_ERROR_LOG: A message is attempting to be sent to a process whose
>> >> contact information is unknown in file rml_oob_send.c at line 105
>> >> <PGP.sig>_______________________________________________
>> >> users mailing list
>> >> users_at_[hidden]
>> >> http://www.open-mpi.org/mailman/listinfo.cgi/users
>> >
>> >
>> > --
>> > Jeff Squyres
>> > Cisco Systems
>> >
>> > _______________________________________________
>> > users mailing list
>> > users_at_[hidden]
>> > http://www.open-mpi.org/mailman/listinfo.cgi/users
>>
>> <PGP.sig>_______________________________________________
>> users mailing list
>> users_at_[hidden]
>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>
>