Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: Re: [OMPI users] using OpenMPI + SGE in a heterogeneous network
From: Reuti (reuti_at_[hidden])
Date: 2008-06-06 19:56:06


Hi,

Am 07.06.2008 um 01:07 schrieb John Hearns:

> On Fri, 2008-06-06 at 17:56 +0100, SLIM H.A. wrote:
>> Hi
>>
>> I want to use SGE to run jobs on a cluster with mx and infiniband
>> nodes.
>> By dividing the nodes into two host groups SGE will submit to either
>> interconnect.
>>
>> The interconnect can be specified in the mpirun command with the --
>> mca
>> btl parameter. However users would then have to decide at runtime
>> which
>> of the two interconnects should be used, but if the job has to queue
>> they just want the first available interconnect.
>
> If you are just using hostgroups, how do you stop SGE from assigning
> jobs across two hostgroups then? IE. if the user does not make an
> explicit request for @infinihosts or @myrihosts SGE will allocate a
> set
> of free hosts - and that could be a mix of the two types.
>
> As Reuti says you need those wildcard PEs.
>
> Or, as you say, just let the user decide when submitting the job.
>
> ps. there is another way of doing the selection, other than
> hostgroups.
> Set up two complexes, infiniband and myrinet (say) which are Boolean.
> Then for each exec host set complex_values infiniband=true
> complex_values myrinet=true
> The user submits with -l infiniband
> I'm not sure if the syntax -l infiniband | myrinet
> meaning infiniband or myrinet is valid. Reuti?

no, it's only for STRING in the complex defintion of 6.1 (RESTRING in
6.0). I.e. you could have a complex INTERCONNECT of type RESTRING and
then request:

-l INTERCONNECT="TCP|OPENIB"

with a queue like:

complex_values NONE,[@group1=INTERCONNECT=OPENIB],
[@group2=INTERCONNECT=TCP]

But this wouldn't prevent getting a mixture either :-/ It would only
allow to select the interconnecttype by name instead of hostgroup. As
the "|" (or) is useless to limit the selection to one of both, you
could even use your BOOL approach to limit to one interface in case
you care about which will be used. Just a matter of taste.

-- Reuti

>
>
>
>
>
> _______________________________________________
> users mailing list
> users_at_[hidden]
> http://www.open-mpi.org/mailman/listinfo.cgi/users