Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: Re: [OMPI users] Hybrid program
From: Gabriele Fatigati (g.fatigati_at_[hidden])
Date: 2008-11-20 10:03:08


And in the hybrid program MPi+OpenMP?
Are these considerations still good?

2008/11/20 Edgar Gabriel <gabriel_at_[hidden]>:
> I don't think that they conflict with our paffinity module and setting. My
> understanding is that if you set a new affinity mask, it simply overwrites
> the previous setting. So in the worst case it voids the setting made by Open
> MPI, but I don't think that it should cause 'problems'. Admittedly, I
> haven't tried the library and the function calls yet, I just learned
> relatively recently about them...
>
> Thanks
> Edga
>
> Ralph Castain wrote:
>>
>> Interesting - learn something new every day! :-)
>>
>> How does this interact with OMPI's paffinity/maffinity assignments? With
>> the rank/slot mapping and binding system?
>>
>> Should users -not- set paffinity if they include these numa calls in their
>> code?
>>
>> Can we detect any potential conflict in OMPI and avoid setting
>> paffinity_alone? Reason I ask: many systems set paffinity_alone in the
>> default mca param file because they always assign dedicated nodes to users.
>> While users can be told to be sure to turn it "off" when using these calls,
>> it seems inevitable that they will forget - and complaints will appear.
>>
>> Thanks
>> Ralph
>>
>>
>>
>> On Nov 20, 2008, at 7:34 AM, Edgar Gabriel wrote:
>>
>>> if you look at recent versions of libnuma, there are two functions called
>>> numa_run_on_node() and numa_run_on_node_mask(), which allow thread-based
>>> assignments to CPUs....
>>>
>>> Thanks
>>> Edgar
>>>
>>> Gabriele Fatigati wrote:
>>>>
>>>> Is there a way to assign one thread to one core? Also from code, not
>>>> necessary with OpenMPI option.
>>>> Thanks.
>>>> 2008/11/19 Stephen Wornom <stephen.wornom_at_[hidden]>:
>>>>>
>>>>> Gabriele Fatigati wrote:
>>>>>>
>>>>>> Ok,
>>>>>> but in Ompi 1.3 how can i enable it?
>>>>>>
>>>>> This may not be relevant, but I could not get a hybrid mpi+OpenMP code
>>>>> to
>>>>> work correctly.
>>>>> Would my problem be related to Gabriele's and perhaps fixed in openmpi
>>>>> 1.3?
>>>>> Stephen
>>>>>>
>>>>>> 2008/11/18 Ralph Castain <rhc_at_[hidden]>:
>>>>>>
>>>>>>> I am afraid it is only available in 1.3 - we didn't backport it to
>>>>>>> the
>>>>>>> 1.2
>>>>>>> series
>>>>>>>
>>>>>>>
>>>>>>> On Nov 18, 2008, at 10:06 AM, Gabriele Fatigati wrote:
>>>>>>>
>>>>>>>
>>>>>>>> Hi,
>>>>>>>> how can i set "slot mapping" as you told me? With TASK GEOMETRY? Or
>>>>>>>> is
>>>>>>>> a new 1.3 OpenMPI feature?
>>>>>>>>
>>>>>>>> Thanks.
>>>>>>>>
>>>>>>>> 2008/11/18 Ralph Castain <rhc_at_[hidden]>:
>>>>>>>>
>>>>>>>>> Unfortunately, paffinity doesn't know anything about assigning
>>>>>>>>> threads
>>>>>>>>> to
>>>>>>>>> cores. This is actually a behavior of Linux, which only allows
>>>>>>>>> paffinity
>>>>>>>>> to
>>>>>>>>> be set at the process level. So, when you set paffinity on a
>>>>>>>>> process,
>>>>>>>>> you
>>>>>>>>> bind all threads of that process to the specified core(s). You
>>>>>>>>> cannot
>>>>>>>>> specify that a thread be given a specific core.
>>>>>>>>>
>>>>>>>>> In this case, your two threads/process are sharing the same core
>>>>>>>>> and
>>>>>>>>> thus
>>>>>>>>> contending for it. As you'd expect in that situation, one thread
>>>>>>>>> gets
>>>>>>>>> the
>>>>>>>>> vast majority of the attention, while the other thread is mostly
>>>>>>>>> idle.
>>>>>>>>>
>>>>>>>>> If you can upgrade to the beta 1.3 release, try using the slot
>>>>>>>>> mapping
>>>>>>>>> to
>>>>>>>>> assign multiple cores to each process. This will ensure that the
>>>>>>>>> threads
>>>>>>>>> for
>>>>>>>>> that process have exclusive access to those cores, but will not
>>>>>>>>> bind a
>>>>>>>>> particular thread to one core - the threads can "move around"
>>>>>>>>> across
>>>>>>>>> the
>>>>>>>>> specified set of cores. Your threads will then be able to run
>>>>>>>>> without
>>>>>>>>> interfering with each other.
>>>>>>>>>
>>>>>>>>> Ralph
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Nov 18, 2008, at 9:18 AM, Gabriele Fatigati wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> Dear OpenMPI developers,
>>>>>>>>>> i have a strange problem with mixed program MPI+OPENMP over
>>>>>>>>>> OpenMPI
>>>>>>>>>> 1.2.6. I'm using PJL TASK GEOMETRY in LSF Scheduler, setting 2
>>>>>>>>>> MPI
>>>>>>>>>> process every compute node, and 2 OMP threads per process. Using
>>>>>>>>>> paffinity and maffinity, i've noted that over every node, i have 2
>>>>>>>>>> thread that works 100%, and 2 threads doesn't works, or works very
>>>>>>>>>> few.
>>>>>>>>>>
>>>>>>>>>> If i disable paffinity and maffinity, 4 threads works well,
>>>>>>>>>> without
>>>>>>>>>> load imbalance.
>>>>>>>>>> I don't understand this issue: paffinity and maffinity should map
>>>>>>>>>> every thread over a specific core, optimizing the cache flow, but
>>>>>>>>>> i
>>>>>>>>>> have this without settings there!
>>>>>>>>>>
>>>>>>>>>> Can i use paffinity and maffinity in mixed MPI+OpenMP program? Or
>>>>>>>>>> it
>>>>>>>>>> works only over MPI thread?
>>>>>>>>>>
>>>>>>>>>> Thanks in advance.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Ing. Gabriele Fatigati
>>>>>>>>>>
>>>>>>>>>> CINECA Systems & Tecnologies Department
>>>>>>>>>>
>>>>>>>>>> Supercomputing Group
>>>>>>>>>>
>>>>>>>>>> Via Magnanelli 6/3, Casalecchio di Reno (BO) Italy
>>>>>>>>>>
>>>>>>>>>> www.cineca.it Tel: +39 051 6171722
>>>>>>>>>>
>>>>>>>>>> g.fatigati_at_[hidden]
>>>>>>>>>> _______________________________________________
>>>>>>>>>> users mailing list
>>>>>>>>>> users_at_[hidden]
>>>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> users mailing list
>>>>>>>>> users_at_[hidden]
>>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>> --
>>>>>>>> Ing. Gabriele Fatigati
>>>>>>>>
>>>>>>>> CINECA Systems & Tecnologies Department
>>>>>>>>
>>>>>>>> Supercomputing Group
>>>>>>>>
>>>>>>>> Via Magnanelli 6/3, Casalecchio di Reno (BO) Italy
>>>>>>>>
>>>>>>>> www.cineca.it Tel: +39 051 6171722
>>>>>>>>
>>>>>>>> g.fatigati_at_[hidden]
>>>>>>>> _______________________________________________
>>>>>>>> users mailing list
>>>>>>>> users_at_[hidden]
>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> users mailing list
>>>>>>> users_at_[hidden]
>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>> stephen.wornom_at_[hidden]
>>>>> 2004 route des lucioles - BP93
>>>>> Sophia Antipolis
>>>>> 06902 CEDEX
>>>>>
>>>>> Tel: 04 92 38 50 54
>>>>> Fax: 04 97 15 53 51
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> users mailing list
>>>>> users_at_[hidden]
>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>>
>>> _______________________________________________
>>> users mailing list
>>> users_at_[hidden]
>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>
>> _______________________________________________
>> users mailing list
>> users_at_[hidden]
>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>
> --
> Edgar Gabriel
> Assistant Professor
> Parallel Software Technologies Lab http://pstl.cs.uh.edu
> Department of Computer Science University of Houston
> Philip G. Hoffman Hall, Room 524 Houston, TX-77204, USA
> Tel: +1 (713) 743-3857 Fax: +1 (713) 743-3335
> _______________________________________________
> users mailing list
> users_at_[hidden]
> http://www.open-mpi.org/mailman/listinfo.cgi/users
>
>

-- 
Ing. Gabriele Fatigati
CINECA Systems & Tecnologies Department
Supercomputing  Group
Via Magnanelli 6/3, Casalecchio di Reno (BO) Italy
www.cineca.it                    Tel:   +39 051 6171722
g.fatigati_at_[hidden]