Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: Re: [OMPI users] Error when using OpenMPI with SGE multiple hosts
From: Ralph Castain (rhc_at_[hidden])
Date: 2010-11-15 09:21:43


I confess I am now confused. What version of OMPI are you using?

FWIW: OMPI was updated at some point to detect the actual cores of an
external binding, and abide by them. If we aren't doing that, then we have a
bug that needs to be resolved. Or it could be you are using a version that
predates the change.

Thanks
Ralph

On Mon, Nov 15, 2010 at 5:38 AM, Reuti <reuti_at_[hidden]> wrote:

> Hi,
>
> Am 15.11.2010 um 13:13 schrieb Chris Jewell:
>
> > Okay so I tried what you suggested. You essentially get the requested
> number of bound cores on each execution node, so if I use
> >
> > $ qsub -pe openmpi 8 -binding linear:2 <myscript.com>
> >
> > then I get 2 bound cores per node, irrespective of the number of slots
> (and hence parallel) processes allocated by GE. This is irrespective of
> which setting I use for the allocation_rule.
>
> but it should work fine with an "allocation_rule 2" then.
>
>
> > My aim with this was to deal with badly behaved multithreaded algorithms
>
> Yep, this causes sometimes the overloading of a machine. When I know that I
> want to compile a parallel Open MPI application, I use non-threaded versions
> of ATLAS, MKL or other libraries.
>
>
> > which end up spreading across more cores on an execution node than the
> number of GE-allocated slots (thereby interfering with other GE scheduled
> tasks running on the same exec node). By binding a process to one or more
> cores, one can "box in" processes and prevent them from spawning erroneous
> sub-processes and threads. Unfortunately, the above solution sets the same
> core binding for each execution node to be the same.
> >
> >> From exploring the software (both OpenMPI and GE) further, I have two
> comments:
> >
> > 1) The core binding feature in GE appears to apply the requested
> core-binding topology to every execution node involved in a parallel job,
> rather than assuming that the topology requested is *per parallel process*.
> So, if I request 'qsub -pe mpi 8 -binding linear:1 <myscript.com>' with
> the intention of getting each of the 8 parallel processes to be bound to 1
> core, I actually get all processes associated with the job_id on one exec
> node bound to 1 core. Oops!
> >
> > 2) OpenMPI has its own core-binding feature (-mca mpi_paffinity_alone 1)
> which works well to bind each parallel process to one processor.
> Unfortunately, the binding framework (hwloc) is different to that which GE
> uses (PLPA), resulting in binding overlaps between GE-bound tasks (eg serial
> and smp jobs) and OpenMPI-bound processes (ie my mpi jobs). Again, oops ;-)
>
> > If, indeed, it is not possible currently to implement this type of
> core-binding in tightly integrated OpenMPI/GE, then a solution might lie in
> a custom script run in the parallel environment's 'start proc args'. This
> script would have to find out which slots are allocated where on the
> cluster, and write an OpenMPI rankfile.
>
> Exactly this should work.
>
> If you use "binding_instance" "pe" and reformat the information in the
> $PE_HOSTFILE to a "rankfile", it should work to get the desired allocation.
> Maybe you can share the script with this list once you got it working.
>
> -- Reuti
> _______________________________________________
> users mailing list
> users_at_[hidden]
> http://www.open-mpi.org/mailman/listinfo.cgi/users
>