Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: Re: [OMPI users] Hybrid program
From: Lenny Verkhovsky (lenny.verkhovsky_at_[hidden])
Date: 2008-11-23 04:06:23


Hi,

Sorry for not answering sooner,

In Open MPI 1.3 we added a paffinity mapping module.

The syntax is quite simple and flexible:

rank N=hostA slot=socket:core_range

rank M=hostB slot=cpu

see the fallowing example:

ex:

#mpirun -rf rankfile_name ./app

#cat rankfile_name

rank 0=host1 slot=0

rank 1=host2 slot=0:*

rank 2=host3 slot=1:0,1

rank 3=host3 slot=1:2-3

rank 4=host1 slot=1:0,0:2

explanation:

Let's assume we have Quad core Dual CPU machines named host1,host2,host3

Using the rankfile above we get rank 0 running on CPU#0 ( cat
/proc/cpu_info you see what is CPU #0 )

rank 1 will run on all cores of socket #0

rank 2 will run on host3 socket #1, cores 0,1

rank 3 will run on host3 socket #1, cores from #2 to #3

rank 4 will run on host1 socket1:core0 and socket0:core2

So, using threads you probably should use slot=0:*, this way all threads
will run on all cores of socket #0 ( or any other specified ).

or coma separated list of exact pairs like rank4 in the example above.

you can also use -mca paffinity_base_verbose 10 to see the mapping that took
place in the job.

Best Regards.

Lenny.

On 11/20/08, Ralph Castain <rhc_at_[hidden]> wrote:
>
> At the very least, you would have to call these functions -after- MPI_Init
> so they could override what OMPI did.
>
>
> On Nov 20, 2008, at 8:03 AM, Gabriele Fatigati wrote:
>
> And in the hybrid program MPi+OpenMP?
>> Are these considerations still good?
>>
>> 2008/11/20 Edgar Gabriel <gabriel_at_[hidden]>:
>>
>>> I don't think that they conflict with our paffinity module and setting.
>>> My
>>> understanding is that if you set a new affinity mask, it simply
>>> overwrites
>>> the previous setting. So in the worst case it voids the setting made by
>>> Open
>>> MPI, but I don't think that it should cause 'problems'. Admittedly, I
>>> haven't tried the library and the function calls yet, I just learned
>>> relatively recently about them...
>>>
>>> Thanks
>>> Edga
>>>
>>> Ralph Castain wrote:
>>>
>>>>
>>>> Interesting - learn something new every day! :-)
>>>>
>>>> How does this interact with OMPI's paffinity/maffinity assignments? With
>>>> the rank/slot mapping and binding system?
>>>>
>>>> Should users -not- set paffinity if they include these numa calls in
>>>> their
>>>> code?
>>>>
>>>> Can we detect any potential conflict in OMPI and avoid setting
>>>> paffinity_alone? Reason I ask: many systems set paffinity_alone in the
>>>> default mca param file because they always assign dedicated nodes to
>>>> users.
>>>> While users can be told to be sure to turn it "off" when using these
>>>> calls,
>>>> it seems inevitable that they will forget - and complaints will appear.
>>>>
>>>> Thanks
>>>> Ralph
>>>>
>>>>
>>>>
>>>> On Nov 20, 2008, at 7:34 AM, Edgar Gabriel wrote:
>>>>
>>>> if you look at recent versions of libnuma, there are two functions
>>>>> called
>>>>> numa_run_on_node() and numa_run_on_node_mask(), which allow
>>>>> thread-based
>>>>> assignments to CPUs....
>>>>>
>>>>> Thanks
>>>>> Edgar
>>>>>
>>>>> Gabriele Fatigati wrote:
>>>>>
>>>>>>
>>>>>> Is there a way to assign one thread to one core? Also from code, not
>>>>>> necessary with OpenMPI option.
>>>>>> Thanks.
>>>>>> 2008/11/19 Stephen Wornom <stephen.wornom_at_[hidden]>:
>>>>>>
>>>>>>>
>>>>>>> Gabriele Fatigati wrote:
>>>>>>>
>>>>>>>>
>>>>>>>> Ok,
>>>>>>>> but in Ompi 1.3 how can i enable it?
>>>>>>>>
>>>>>>>> This may not be relevant, but I could not get a hybrid mpi+OpenMP
>>>>>>> code
>>>>>>> to
>>>>>>> work correctly.
>>>>>>> Would my problem be related to Gabriele's and perhaps fixed in
>>>>>>> openmpi
>>>>>>> 1.3?
>>>>>>> Stephen
>>>>>>>
>>>>>>>>
>>>>>>>> 2008/11/18 Ralph Castain <rhc_at_[hidden]>:
>>>>>>>>
>>>>>>>> I am afraid it is only available in 1.3 - we didn't backport it to
>>>>>>>>> the
>>>>>>>>> 1.2
>>>>>>>>> series
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Nov 18, 2008, at 10:06 AM, Gabriele Fatigati wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Hi,
>>>>>>>>>> how can i set "slot mapping" as you told me? With TASK GEOMETRY?
>>>>>>>>>> Or
>>>>>>>>>> is
>>>>>>>>>> a new 1.3 OpenMPI feature?
>>>>>>>>>>
>>>>>>>>>> Thanks.
>>>>>>>>>>
>>>>>>>>>> 2008/11/18 Ralph Castain <rhc_at_[hidden]>:
>>>>>>>>>>
>>>>>>>>>> Unfortunately, paffinity doesn't know anything about assigning
>>>>>>>>>>> threads
>>>>>>>>>>> to
>>>>>>>>>>> cores. This is actually a behavior of Linux, which only allows
>>>>>>>>>>> paffinity
>>>>>>>>>>> to
>>>>>>>>>>> be set at the process level. So, when you set paffinity on a
>>>>>>>>>>> process,
>>>>>>>>>>> you
>>>>>>>>>>> bind all threads of that process to the specified core(s). You
>>>>>>>>>>> cannot
>>>>>>>>>>> specify that a thread be given a specific core.
>>>>>>>>>>>
>>>>>>>>>>> In this case, your two threads/process are sharing the same core
>>>>>>>>>>> and
>>>>>>>>>>> thus
>>>>>>>>>>> contending for it. As you'd expect in that situation, one thread
>>>>>>>>>>> gets
>>>>>>>>>>> the
>>>>>>>>>>> vast majority of the attention, while the other thread is mostly
>>>>>>>>>>> idle.
>>>>>>>>>>>
>>>>>>>>>>> If you can upgrade to the beta 1.3 release, try using the slot
>>>>>>>>>>> mapping
>>>>>>>>>>> to
>>>>>>>>>>> assign multiple cores to each process. This will ensure that the
>>>>>>>>>>> threads
>>>>>>>>>>> for
>>>>>>>>>>> that process have exclusive access to those cores, but will not
>>>>>>>>>>> bind a
>>>>>>>>>>> particular thread to one core - the threads can "move around"
>>>>>>>>>>> across
>>>>>>>>>>> the
>>>>>>>>>>> specified set of cores. Your threads will then be able to run
>>>>>>>>>>> without
>>>>>>>>>>> interfering with each other.
>>>>>>>>>>>
>>>>>>>>>>> Ralph
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Nov 18, 2008, at 9:18 AM, Gabriele Fatigati wrote:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Dear OpenMPI developers,
>>>>>>>>>>>> i have a strange problem with mixed program MPI+OPENMP over
>>>>>>>>>>>> OpenMPI
>>>>>>>>>>>> 1.2.6. I'm using PJL TASK GEOMETRY in LSF Scheduler, setting 2
>>>>>>>>>>>> MPI
>>>>>>>>>>>> process every compute node, and 2 OMP threads per process. Using
>>>>>>>>>>>> paffinity and maffinity, i've noted that over every node, i have
>>>>>>>>>>>> 2
>>>>>>>>>>>> thread that works 100%, and 2 threads doesn't works, or works
>>>>>>>>>>>> very
>>>>>>>>>>>> few.
>>>>>>>>>>>>
>>>>>>>>>>>> If i disable paffinity and maffinity, 4 threads works well,
>>>>>>>>>>>> without
>>>>>>>>>>>> load imbalance.
>>>>>>>>>>>> I don't understand this issue: paffinity and maffinity should
>>>>>>>>>>>> map
>>>>>>>>>>>> every thread over a specific core, optimizing the cache flow,
>>>>>>>>>>>> but
>>>>>>>>>>>> i
>>>>>>>>>>>> have this without settings there!
>>>>>>>>>>>>
>>>>>>>>>>>> Can i use paffinity and maffinity in mixed MPI+OpenMP program?
>>>>>>>>>>>> Or
>>>>>>>>>>>> it
>>>>>>>>>>>> works only over MPI thread?
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks in advance.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> Ing. Gabriele Fatigati
>>>>>>>>>>>>
>>>>>>>>>>>> CINECA Systems & Tecnologies Department
>>>>>>>>>>>>
>>>>>>>>>>>> Supercomputing Group
>>>>>>>>>>>>
>>>>>>>>>>>> Via Magnanelli 6/3, Casalecchio di Reno (BO) Italy
>>>>>>>>>>>>
>>>>>>>>>>>> www.cineca.it Tel: +39 051 6171722
>>>>>>>>>>>>
>>>>>>>>>>>> g.fatigati_at_[hidden]
>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>> users mailing list
>>>>>>>>>>>> users_at_[hidden]
>>>>>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>>>>>>>>>
>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>> users mailing list
>>>>>>>>>>> users_at_[hidden]
>>>>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>> Ing. Gabriele Fatigati
>>>>>>>>>>
>>>>>>>>>> CINECA Systems & Tecnologies Department
>>>>>>>>>>
>>>>>>>>>> Supercomputing Group
>>>>>>>>>>
>>>>>>>>>> Via Magnanelli 6/3, Casalecchio di Reno (BO) Italy
>>>>>>>>>>
>>>>>>>>>> www.cineca.it Tel: +39 051 6171722
>>>>>>>>>>
>>>>>>>>>> g.fatigati_at_[hidden]
>>>>>>>>>> _______________________________________________
>>>>>>>>>> users mailing list
>>>>>>>>>> users_at_[hidden]
>>>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>>>>>>>
>>>>>>>>>> _______________________________________________
>>>>>>>>> users mailing list
>>>>>>>>> users_at_[hidden]
>>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>> --
>>>>>>> stephen.wornom_at_[hidden]
>>>>>>> 2004 route des lucioles - BP93
>>>>>>> Sophia Antipolis
>>>>>>> 06902 CEDEX
>>>>>>>
>>>>>>> Tel: 04 92 38 50 54
>>>>>>> Fax: 04 97 15 53 51
>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> users mailing list
>>>>>>> users_at_[hidden]
>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>>>>
>>>>>>> _______________________________________________
>>>>> users mailing list
>>>>> users_at_[hidden]
>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>>
>>>>
>>>> _______________________________________________
>>>> users mailing list
>>>> users_at_[hidden]
>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>
>>>
>>> --
>>> Edgar Gabriel
>>> Assistant Professor
>>> Parallel Software Technologies Lab http://pstl.cs.uh.edu
>>> Department of Computer Science University of Houston
>>> Philip G. Hoffman Hall, Room 524 Houston, TX-77204, USA
>>> Tel: +1 (713) 743-3857 Fax: +1 (713) 743-3335
>>> _______________________________________________
>>> users mailing list
>>> users_at_[hidden]
>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>
>>>
>>>
>>
>>
>> --
>> Ing. Gabriele Fatigati
>>
>> CINECA Systems & Tecnologies Department
>>
>> Supercomputing Group
>>
>> Via Magnanelli 6/3, Casalecchio di Reno (BO) Italy
>>
>> www.cineca.it Tel: +39 051 6171722
>>
>> g.fatigati_at_[hidden]
>> _______________________________________________
>> users mailing list
>> users_at_[hidden]
>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>
>
> _______________________________________________
> users mailing list
> users_at_[hidden]
> http://www.open-mpi.org/mailman/listinfo.cgi/users
>