Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: Re: [OMPI users] Mixed Threaded MPI code, how to launch?
From: Ralph Castain (rhc_at_[hidden])
Date: 2008-10-30 14:49:42


Brock

I have a patch for 1.2 that will allow hostfiles to work with TM. If
it would help, I can send it to you off-list.

Ralph

On Oct 30, 2008, at 8:40 AM, Brock Palen wrote:

> Yes I never made it to my mailbox. Strange, (wink wink, ahh email).
>
> Thanks for letting me know about it, I have the message now.
>
> as for using 1.3 prerelease, that is not really an option right now
> for us. I think we can get by with 1.2 without threads or do some
> hacking (ppn=largest number we have launch with -bynode).
> TIll a 1.3 stable is out.
>
> Thanks, new features for launching look really neat.
>
> Brock Palen
> www.umich.edu/~brockp
> Center for Advanced Computing
> brockp_at_[hidden]
> (734)936-1985
>
>
>
> On Oct 30, 2008, at 10:12 AM, Ralph Castain wrote:
>
>> I believe I answered much of this the other day - did it get lost
>> in the email?
>>
>> As for using TM with a hostfile - this is an unfortunately bug in
>> the 1.2 series. You can't - you'll have to move to 1.3 to do so.
>> When you do, note the changed handling of hostfiles as specified on
>> the wiki:
>>
>> https://svn.open-mpi.org/trac/ompi/wiki/HostFilePlan
>>
>> Ralph
>>
>>
>>> I take it this is using OMPI 1.2.x? If so, there really isn't a
>>> way to do this in that series.
>>>
>>> If they are using 1.3 (in some pre-release form), then there are
>>> two options:
>>>
>>> 1. they could use the sequential mapper by specifying "-mca rmaps
>>> seq". This mapper takes a hostfile and maps one process to each
>>> entry, in rank order. So they could specify that we only map to
>>> half of the actual number of cores on a particular node
>>>
>>> 2. they could use the rank_file mapper that allows you to specify
>>> what cores are to be used by what rank. I am less familiar with
>>> this option and there isn't a lot of documentation on how to use
>>> it - but you may have to provide a fairly comprehensive map file
>>> since your nodes are not all the same.
>>>
>>> I have been asked by some other folks to provide a mapping option
>>> "--stride x" that would cause the default round-robin mapper to
>>> step across the specified number of slots. So a stride of 2 would
>>> automatically cause byslot mapping to increment by 2 instead of
>>> the current stride of 1. I doubt that will be in 1.3.0, but it
>>> will show up in later releases.
>>>
>>> Ralph
>>>
>> On Oct 30, 2008, at 7:46 AM, Brock Palen wrote:
>>
>>> Any thoughts on this?
>>>
>>> We are looking writing a script that parses $PBS_NODEFILE to
>>> create a machinefile and using -machinefile
>>>
>>> When we do that though we have to disable tm to avoid an error (-
>>> mca pls ^tm) this is far from preferable.
>>>
>>> Any ideas to tell mpirun to only launch on half the cpus given to
>>> it by PBS, but each cpu must have adjacent to it another cpu in
>>> the same node?
>>>
>>> Brock Palen
>>> www.umich.edu/~brockp
>>> Center for Advanced Computing
>>> brockp_at_[hidden]
>>> (734)936-1985
>>>
>>>
>>>
>>> On Oct 25, 2008, at 5:36 PM, Brock Palen wrote:
>>>
>>>> We have a user with a code that uses threaded solvers inside each
>>>> MPI rank. They would like to run two threads per process.
>>>>
>>>> The question is how to launch this? The default -byslot puts all
>>>> the processes on the first sets of cpus not leaving any cpus for
>>>> the second thread for each process. And half the cpus are wasted.
>>>>
>>>> The -bynode option works in theory, if all our nodes had the same
>>>> number of core (they do not).
>>>>
>>>> So right now the user did:
>>>>
>>>> #PBS -l nodes=22:ppn=2
>>>> export OMP_NUM_THREADS=2
>>>> mpirun -np 22 app
>>>>
>>>> Which made me aware of the problem.
>>>>
>>>> How can I basically tell OMPI that a 'slot' is two cores on the
>>>> same machine? This needs to work inside out torque based
>>>> queueing system.
>>>>
>>>> Sorry If I was not clear about my goal.
>>>>
>>>>
>>>> Brock Palen
>>>> www.umich.edu/~brockp
>>>> Center for Advanced Computing
>>>> brockp_at_[hidden]
>>>> (734)936-1985
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> users mailing list
>>>> users_at_[hidden]
>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>
>>>>
>>>
>>> _______________________________________________
>>> users mailing list
>>> users_at_[hidden]
>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>
>> _______________________________________________
>> users mailing list
>> users_at_[hidden]
>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>
>>
>
> _______________________________________________
> users mailing list
> users_at_[hidden]
> http://www.open-mpi.org/mailman/listinfo.cgi/users