Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

From: Ralph Castain (rhc_at_[hidden])
Date: 2007-02-02 17:20:06


Hi Todd

To help us provide advice, could you tell us what version of OpenMPI you are
using?

Meantime, try adding ³-mca pls_rsh_num_concurrent 200² to your mpirun
command line. You can up the number of concurrent daemons we launch to
anything your system will support ­ basically, we limit the number only
because some systems have limits on the number of ssh calls we can have
active at any one time. Because we hold stdio open when running with
‹debug-daemons, the number of concurrent daemons must match or exceed the
number of nodes you are trying to launch on.

I have a ³fix² in the works that will help relieve some of that restriction,
but that won¹t come out until a later release.

Hopefully, that will allow you to obtain more debug info about why/where
things are hanging.

Ralph

On 2/2/07 11:41 AM, "Heywood, Todd" <heywood_at_[hidden]> wrote:

> I have OpenMPI running fine for a small/medium number of tasks (simple hello
> or cpi program). But when I try 700 or 800 tasks, it hangs, apparently on
> startup. I think this might be related to LDAP, since if I try to log into my
> account while the job is hung, I get told my username doesn¹t exist. However,
> I tried adding ­debug to the mpirun, and got the same sequence of output as
> for successful smaller runs, until it hung again. So I added ­-debug-daemons
> and got this (with an exit, i.e. no hanging):
> Š
> [blade1:31733] [0,0,0] wrote setup file
> --------------------------------------------------------------------------
> The rsh launcher has been given a number of 128 concurrent daemons to
> launch and is in a debug-daemons option. However, the total number of
> daemons to launch (200) is greater than this value. This is a scenario that
> will cause the system to deadlock.
>
> To avoid deadlock, either increase the number of concurrent daemons, or
> remove the debug-daemons flag.
> --------------------------------------------------------------------------
> [blade1:31733] [0,0,0] ORTE_ERROR_LOG: Fatal in file
> ../../../../../orte/mca/rmgr/urm/
> rmgr_urm.c at line 455
> [blade1:31733] mpirun: spawn failed with errno=-6
> [blade1:31733] sess_dir_finalize: proc session dir not empty - leaving
>
> Any ideas or suggestions appreciated.
>
> Todd Heywood
>
>
>
>
>
>
> _______________________________________________
> users mailing list
> users_at_[hidden]
> http://www.open-mpi.org/mailman/listinfo.cgi/users