Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: Re: [OMPI users] Hybrid program
From: Ralph Castain (rhc_at_[hidden])
Date: 2008-11-20 10:26:53


At the very least, you would have to call these functions -after-
MPI_Init so they could override what OMPI did.

On Nov 20, 2008, at 8:03 AM, Gabriele Fatigati wrote:

> And in the hybrid program MPi+OpenMP?
> Are these considerations still good?
>
> 2008/11/20 Edgar Gabriel <gabriel_at_[hidden]>:
>> I don't think that they conflict with our paffinity module and
>> setting. My
>> understanding is that if you set a new affinity mask, it simply
>> overwrites
>> the previous setting. So in the worst case it voids the setting
>> made by Open
>> MPI, but I don't think that it should cause 'problems'. Admittedly, I
>> haven't tried the library and the function calls yet, I just learned
>> relatively recently about them...
>>
>> Thanks
>> Edga
>>
>> Ralph Castain wrote:
>>>
>>> Interesting - learn something new every day! :-)
>>>
>>> How does this interact with OMPI's paffinity/maffinity
>>> assignments? With
>>> the rank/slot mapping and binding system?
>>>
>>> Should users -not- set paffinity if they include these numa calls
>>> in their
>>> code?
>>>
>>> Can we detect any potential conflict in OMPI and avoid setting
>>> paffinity_alone? Reason I ask: many systems set paffinity_alone in
>>> the
>>> default mca param file because they always assign dedicated nodes
>>> to users.
>>> While users can be told to be sure to turn it "off" when using
>>> these calls,
>>> it seems inevitable that they will forget - and complaints will
>>> appear.
>>>
>>> Thanks
>>> Ralph
>>>
>>>
>>>
>>> On Nov 20, 2008, at 7:34 AM, Edgar Gabriel wrote:
>>>
>>>> if you look at recent versions of libnuma, there are two
>>>> functions called
>>>> numa_run_on_node() and numa_run_on_node_mask(), which allow
>>>> thread-based
>>>> assignments to CPUs....
>>>>
>>>> Thanks
>>>> Edgar
>>>>
>>>> Gabriele Fatigati wrote:
>>>>>
>>>>> Is there a way to assign one thread to one core? Also from code,
>>>>> not
>>>>> necessary with OpenMPI option.
>>>>> Thanks.
>>>>> 2008/11/19 Stephen Wornom <stephen.wornom_at_[hidden]>:
>>>>>>
>>>>>> Gabriele Fatigati wrote:
>>>>>>>
>>>>>>> Ok,
>>>>>>> but in Ompi 1.3 how can i enable it?
>>>>>>>
>>>>>> This may not be relevant, but I could not get a hybrid mpi
>>>>>> +OpenMP code
>>>>>> to
>>>>>> work correctly.
>>>>>> Would my problem be related to Gabriele's and perhaps fixed in
>>>>>> openmpi
>>>>>> 1.3?
>>>>>> Stephen
>>>>>>>
>>>>>>> 2008/11/18 Ralph Castain <rhc_at_[hidden]>:
>>>>>>>
>>>>>>>> I am afraid it is only available in 1.3 - we didn't backport
>>>>>>>> it to
>>>>>>>> the
>>>>>>>> 1.2
>>>>>>>> series
>>>>>>>>
>>>>>>>>
>>>>>>>> On Nov 18, 2008, at 10:06 AM, Gabriele Fatigati wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>>> Hi,
>>>>>>>>> how can i set "slot mapping" as you told me? With TASK
>>>>>>>>> GEOMETRY? Or
>>>>>>>>> is
>>>>>>>>> a new 1.3 OpenMPI feature?
>>>>>>>>>
>>>>>>>>> Thanks.
>>>>>>>>>
>>>>>>>>> 2008/11/18 Ralph Castain <rhc_at_[hidden]>:
>>>>>>>>>
>>>>>>>>>> Unfortunately, paffinity doesn't know anything about
>>>>>>>>>> assigning
>>>>>>>>>> threads
>>>>>>>>>> to
>>>>>>>>>> cores. This is actually a behavior of Linux, which only
>>>>>>>>>> allows
>>>>>>>>>> paffinity
>>>>>>>>>> to
>>>>>>>>>> be set at the process level. So, when you set paffinity on a
>>>>>>>>>> process,
>>>>>>>>>> you
>>>>>>>>>> bind all threads of that process to the specified core(s).
>>>>>>>>>> You
>>>>>>>>>> cannot
>>>>>>>>>> specify that a thread be given a specific core.
>>>>>>>>>>
>>>>>>>>>> In this case, your two threads/process are sharing the same
>>>>>>>>>> core
>>>>>>>>>> and
>>>>>>>>>> thus
>>>>>>>>>> contending for it. As you'd expect in that situation, one
>>>>>>>>>> thread
>>>>>>>>>> gets
>>>>>>>>>> the
>>>>>>>>>> vast majority of the attention, while the other thread is
>>>>>>>>>> mostly
>>>>>>>>>> idle.
>>>>>>>>>>
>>>>>>>>>> If you can upgrade to the beta 1.3 release, try using the
>>>>>>>>>> slot
>>>>>>>>>> mapping
>>>>>>>>>> to
>>>>>>>>>> assign multiple cores to each process. This will ensure
>>>>>>>>>> that the
>>>>>>>>>> threads
>>>>>>>>>> for
>>>>>>>>>> that process have exclusive access to those cores, but will
>>>>>>>>>> not
>>>>>>>>>> bind a
>>>>>>>>>> particular thread to one core - the threads can "move around"
>>>>>>>>>> across
>>>>>>>>>> the
>>>>>>>>>> specified set of cores. Your threads will then be able to run
>>>>>>>>>> without
>>>>>>>>>> interfering with each other.
>>>>>>>>>>
>>>>>>>>>> Ralph
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Nov 18, 2008, at 9:18 AM, Gabriele Fatigati wrote:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> Dear OpenMPI developers,
>>>>>>>>>>> i have a strange problem with mixed program MPI+OPENMP over
>>>>>>>>>>> OpenMPI
>>>>>>>>>>> 1.2.6. I'm using PJL TASK GEOMETRY in LSF Scheduler,
>>>>>>>>>>> setting 2
>>>>>>>>>>> MPI
>>>>>>>>>>> process every compute node, and 2 OMP threads per process.
>>>>>>>>>>> Using
>>>>>>>>>>> paffinity and maffinity, i've noted that over every node,
>>>>>>>>>>> i have 2
>>>>>>>>>>> thread that works 100%, and 2 threads doesn't works, or
>>>>>>>>>>> works very
>>>>>>>>>>> few.
>>>>>>>>>>>
>>>>>>>>>>> If i disable paffinity and maffinity, 4 threads works well,
>>>>>>>>>>> without
>>>>>>>>>>> load imbalance.
>>>>>>>>>>> I don't understand this issue: paffinity and maffinity
>>>>>>>>>>> should map
>>>>>>>>>>> every thread over a specific core, optimizing the cache
>>>>>>>>>>> flow, but
>>>>>>>>>>> i
>>>>>>>>>>> have this without settings there!
>>>>>>>>>>>
>>>>>>>>>>> Can i use paffinity and maffinity in mixed MPI+OpenMP
>>>>>>>>>>> program? Or
>>>>>>>>>>> it
>>>>>>>>>>> works only over MPI thread?
>>>>>>>>>>>
>>>>>>>>>>> Thanks in advance.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Ing. Gabriele Fatigati
>>>>>>>>>>>
>>>>>>>>>>> CINECA Systems & Tecnologies Department
>>>>>>>>>>>
>>>>>>>>>>> Supercomputing Group
>>>>>>>>>>>
>>>>>>>>>>> Via Magnanelli 6/3, Casalecchio di Reno (BO) Italy
>>>>>>>>>>>
>>>>>>>>>>> www.cineca.it Tel: +39 051 6171722
>>>>>>>>>>>
>>>>>>>>>>> g.fatigati_at_[hidden]
>>>>>>>>>>> _______________________________________________
>>>>>>>>>>> users mailing list
>>>>>>>>>>> users_at_[hidden]
>>>>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>>>>>>>>
>>>>>>>>>> _______________________________________________
>>>>>>>>>> users mailing list
>>>>>>>>>> users_at_[hidden]
>>>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Ing. Gabriele Fatigati
>>>>>>>>>
>>>>>>>>> CINECA Systems & Tecnologies Department
>>>>>>>>>
>>>>>>>>> Supercomputing Group
>>>>>>>>>
>>>>>>>>> Via Magnanelli 6/3, Casalecchio di Reno (BO) Italy
>>>>>>>>>
>>>>>>>>> www.cineca.it Tel: +39 051 6171722
>>>>>>>>>
>>>>>>>>> g.fatigati_at_[hidden]
>>>>>>>>> _______________________________________________
>>>>>>>>> users mailing list
>>>>>>>>> users_at_[hidden]
>>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> users mailing list
>>>>>>>> users_at_[hidden]
>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> --
>>>>>> stephen.wornom_at_[hidden]
>>>>>> 2004 route des lucioles - BP93
>>>>>> Sophia Antipolis
>>>>>> 06902 CEDEX
>>>>>>
>>>>>> Tel: 04 92 38 50 54
>>>>>> Fax: 04 97 15 53 51
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> users mailing list
>>>>>> users_at_[hidden]
>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>>>
>>>> _______________________________________________
>>>> users mailing list
>>>> users_at_[hidden]
>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>
>>> _______________________________________________
>>> users mailing list
>>> users_at_[hidden]
>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>
>> --
>> Edgar Gabriel
>> Assistant Professor
>> Parallel Software Technologies Lab http://pstl.cs.uh.edu
>> Department of Computer Science University of Houston
>> Philip G. Hoffman Hall, Room 524 Houston, TX-77204, USA
>> Tel: +1 (713) 743-3857 Fax: +1 (713) 743-3335
>> _______________________________________________
>> users mailing list
>> users_at_[hidden]
>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>
>>
>
>
>
> --
> Ing. Gabriele Fatigati
>
> CINECA Systems & Tecnologies Department
>
> Supercomputing Group
>
> Via Magnanelli 6/3, Casalecchio di Reno (BO) Italy
>
> www.cineca.it Tel: +39 051 6171722
>
> g.fatigati_at_[hidden]
> _______________________________________________
> users mailing list
> users_at_[hidden]
> http://www.open-mpi.org/mailman/listinfo.cgi/users