Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: Re: [OMPI users] Error when using OpenMPI with SGE multiple hosts
From: Reuti (reuti_at_[hidden])
Date: 2010-10-14 08:04:05


Hi,

Am 14.10.2010 um 13:23 schrieb Dave Love:

> Reuti <reuti_at_[hidden]> writes:
>
>> With the default binding_instance set to "set" (the default) the
>> shepherd should bind the processes to cores already. With other types
>> of binding_instance these selected cores must be forward to the
>> application via an environment variable or in the hostfile.
>
> My question was specifically about SGE/OMPI tight integration; are you
> actually doing binding successfully with that? I think I read here that
> the integration doesn't (yet?) deal with SGE core binding, and when we
> turned on the SGE feature we got the OMPI tasks piled onto a single
> core. We quickly turned it off for MPI jobs when we realized what was
> happening, and I didn't try to investigate further.

what did you request in particular in `qsub -binding`? When you request `qsub -pe openmpi 2 -binding linear:1 ...` it would apply the core assignment per `qrsh`. Means, when you are staying on one machine only (because of "$pe_slots" for "allocation_rule"), you would indeed oversubscribe the core as Open MPI will then use threads (hence "-binding linear:2" should do in this case). But if the "allocation_rule" is set to the integer value "1" and you get for sure a core on another machine, then "linear:1" would be fine. Similar `qsub -pe openmpi 4 -binding linear:2 ...` when you have an "allocation_rule" of "2".

If in a similar scenario you get 4 cores on one and the same machine and SGE creates a cpuset of 4 cores, these 4 threads can nevertheless be scheduled to any granted core by the Linux scheduler kernel. It would be necssary to use another binding_instance "env" or "pe" to get the information of granted cores into the jobscript/hostfile and decide on your own how to forward this to Open MPI to have each thread also bound to a unique core too and avoid having them drifting around the cores in the cpuset.

-- Reuti

>> As this is only a hint to SGE and not a hard request, the user must
>> plan a little bit the allocation beforehand. Especially if you
>> oversubscribe a machine it won't work.
>
> [It is documented that the binding isn't applied if the selected cores
> are occupied.]
>
> _______________________________________________
> users mailing list
> users_at_[hidden]
> http://www.open-mpi.org/mailman/listinfo.cgi/users