Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: [OMPI users] OpenMPI and SGE
From: Ray Muno (muno_at_[hidden])
Date: 2009-06-23 10:02:07


We are running a cluster using Rocks 5.0 and OpenMPI 1.2 (primarily).
Scheduling is done through SGE. MPI communication is over InfiniBand.

We have been running with this setup for over 9 months. Last week, all
user jobs stopped executing (cluster load dropped to zero). User can
schedule jobs but when they try to execute, they get errors of the form:

-----------------------------------------
[compute-2-5.local:12321] ERROR: The daemon exited unexpectedly with
status 1.
error: commlib error: access denied (client IP resolved to host name "".
This is not identical to clients host name "")
error: executing task of job 11901 failed: failed sending task to
execd_at_compute-5-9.local: can't find connection
[compute-2-5.local:12321] ERROR: A daemon on node compute-5-9.local
failed to start as expected.
[compute-2-5.local:12321] ERROR: There may be more information available
from
[compute-2-5.local:12321] ERROR: the 'qstat -t' command on the Grid
Engine tasks.
[compute-2-5.local:12321] ERROR: If the problem persists, please restart
the
[compute-2-5.local:12321] ERROR: Grid Engine PE job
[compute-2-5.local:12321] ERROR: The daemon exited unexpectedly with
status 1.
error: commlib error: access denied (client IP resolved to host name "".
This is not identical to clients host name "")

When run interactively, we see.

---------------------------------------------
error: commlib error: access denied (client IP resolved to host name "".
This is not identical to clients host name "")
error: executing task of job 12094 failed: failed sending task to
execd_at_compute-4-11.local: can't find connection
--------------------------------------------------------------------------
A daemon (pid 4938) died unexpectedly with status 1 while attempting
to launch so we are aborting.

There may be more information reported by the environment (see above).

This may be because the daemon was unable to find all the needed shared
libraries on the remote node. You may set your LD_LIBRARY_PATH to have the
location of the shared libraries on the remote nodes and this will
automatically be forwarded to the remote nodes.
--------------------------------------------------------------------------
mpirun noticed that the job aborted, but has no info as to the process
that caused that situation.
--------------------------------------------------------------------------
mpirun: clean termination accomplished
---------------------------------------------

This seems to be an error with SGE but it is only affecting OpenMPI.
User can successfully launch and run jobs with MVAPICH.

Some changes were made to the ROCKS setup that may have caused this but
I have not found where the actual problems lies.

-- 
 Ray Muno
 University of Minnesota