Geoff Galitz wrote:
> On Jan 24, 2007, at 7:03 AM, Pak Lui wrote:
>> Geoff Galitz wrote:
>>> On the following system:
>>> OpenMPI 1.1.1
>>> SGE 6.0 (with tight integration)
>>> Scientific Linux 4.3
>>> Dual Dual-Core Opterons
>>> MPI jobs are oversubscribing to the nodes. No matter where jobs
>>> are launched by the scheduler, they always stack up on the first
>>> node (node00) and continue to stack even though the system load
>>> exceeds 6 (on a 4 processor box). Eeach node is defined as 4
>>> slots with 4 max slots. The MPI jobs launch via "mpirun -np
>>> (some-number-of- processors)" from within the scheduler.
>> Hi Geoff,
>> I think we first start having SGE support in 1.2, not in 1.1.1.
>> Unless you did some modification on your own to include the
>> gridengine ras/pls modules from v1.2, you probably are not using
>> the SGE tight integration. So even though you start mpirun in the
>> SGE parallel environment, ORTE does not have the gridengine modules
>> for allocating and launching the jobs, so that could be why all
>> processes are launched on the same node. (because there's no node
>> list available from gridengine and it defaults to single node)
> I have used the backport instructions provided by Olli-Pekka Lehto.
> Of course, if it is running properly in my case, I can't say as I am
> certainly not getting the expected behavior, although the jobs do run.
There are a few things you can try to validate if the gridengine plugin
is being used. First, you might want to check ompi_info to see if
'gridengine' is in the list. Another thing is to use the mpirun -d flag
to show which ras and pls component that mpirun is running through.
There are also a few of MCA params you can use to show when the
gridengine ras+pls modules are in action. Here's a couple:
"-mca ras_gridengine_verbose 1" would show the output from SGE as if you
are passing in the -verbose flag to qrsh.
"-mca pls_gridengine_debug 1" would show you the qrsh -inherit command
that is used to send the job to SGE.
If you are not able to see those outputs, chances are mpirun is not
using the gridengine modules. And that might be a problem with your
setup. I would suggest you to either verify the build process, or try
out the v1.2 instead.
>> On a related note, there is a way for SGE to allocate and assign
>> slots for launching tasks. It is done by setting the allocation
>> rule in the parallel environment (PE). If all of the slots are
>> allocated on the same node, it sounds like the allocation rule has
>> been set to $fill_up. Maybe you can try with $round_robin instead?
> If I use $round_robin, one MPI process starts up per node and then
> wraps around the cluster. So if I have 4 process MPI job, it starts
> 1 process on 4 nodes which is certainly not the most efficient method.
>>> It seems to me that MPI is not detecting that the nodes are
>>> overloaded and that due to the way the job slots are defined and
>>> how mpirun is being called. If I read the documentation
>>> correctly, a single mpirun run consumes one job slot no matter
>>> the number of processes which are launched. We can chagne the
>>> number of job slots, but then we expect to waste processors since
>>> only one mpirun job will run on any node, even if the job is only
>>> a two processor job.
>> As for oversubscription, I remember we start having that -
>> nooversubscribe option in v1.2 so if you want to limit ORTE from
>> oversubscribing because by default oversubscription is allowed.
> So it seems the real story for me is that there is no logic that
> detects the oversubscription condition and re-schedules the job for
> another node in the MPI nodelist in OpenMPI 1.1.1? If so, that would
> certainly explain what I am seeing. Is that correct?
Actually I take back the comment about v1.1.1 doesn't have the
-oversubscription option. It just check the source and it's there in
orterun, so there is that option available to prevent oversubscription.
The behavior you are seeing is probably due the resource allocation
(RAS) not getting the nodelist (from SGE), so RMAPS does not have the
nodelist to map the processes to, and therefore it's using the same node
to launch the user processes.
> users mailing list
- Pak Lui