Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: Re: [OMPI users] Hybrid program
From: Jeff Squyres (jsquyres_at_[hidden])
Date: 2008-12-05 10:52:20


Nifty -- good to know. Thanks for looking into this!

Do any kernel-hacker types on this list know roundabout what version
thread-affinity was brought into the Linux kernel?

FWIW: all the same concepts here (using pid==0) should also work for
PLPA, so you can set via socket/core, etc.

On Dec 5, 2008, at 10:45 AM, Edgar Gabriel wrote:

> ok, so I digged a little deeper, and have some good news. Let me
> start with a set of routines, that we didn't even discuss yet, but
> which works for setting thread affinity, and discuss then libnuma
> and sched_setaffinity() again.
>
> -----------
> On linux systems, the pthread library has a set of routines to
> modify and determine thread-affinity related information:
>
> #define __USE_GNU
>
> int pthread_setaffinity_np (pthread_t __th, size_t __cpusetsize,
> const cpu_set_t *__cpuset);
> int pthread_getaffinity_np (pthread_t __th, size_t __cpusetsize,
> cpu_set_t *__cpuset)
>
> These two routines can be used to modify the affinity of an existing
> thread. If you would like to modify the affinity of a thread
> *before* creating it, you can use a similar routines.
>
> int pthread_attr_setaffinity_np (pthread_attr_t *__attr,
> size_t __cpusetsize,
> const cpu_set_t *__cpuset)
>
> I tested the first two routines, and they did work for me.
> -----------
>
> Now to libnuma vs. sched_setaffinity: after digging a little deeper
> in the libnuma sources, I realized that one of the differences on
> what they do vs. what I did in my testcases was, that libnuma uses
> the sched_setaffinity() calls with a pid of 0, instead of
> determining the pid using the getpid() function. According to the
> sched_setaffinity() manpages, pid of zero means 'apply the new rules
> to the current process', but it does in fact mean 'to the current
> task/thread'. I wrote a set of tests, where I used
> sched_setaffinity() with the pid zero, and I was in fact able to
> modify the affinity of an individual thread using
> sched_setaffinity(). If you pass in the pid of the process, it will
> affect the affinity of all threads of that process.
>
> Bottom line is, you can modify the affinity of a thread using both
> libnuma on a per socket basis and the sched_setaffinity() calls on a
> per core basis. Alternatively, you can use the
> pthread_setaffinity_np() function to modify the affinity of a thread
> using a cpu_set_t similar to sched_setaffinity.
>
> Thanks
> Edgar
>
>
> Jeff Squyres wrote:
>> Fair enough; let me know what you find. It would be good to
>> understand exactly why you're seeing what you're seeing...
>> On Dec 2, 2008, at 5:47 PM, Edgar Gabriel wrote:
>>> its on OpenSuSE 11 with kernel 2.6.25.11. I don't know the libnuma
>>> library version, but I suspect that its fairly new.
>>>
>>> I will try to investigate that in the next days a little more. I
>>> do think that they use sched_setaffinity() underneath the hood
>>> (because in one of my failed attempts when I passed in the wrong
>>> argument, I got actually the same error message that I got earlier
>>> with sched_setaffinity), but they must do something additionally
>>> underneath.
>>>
>>> Anyway, I just wanted to report the result, and that there is
>>> obviously a difference, even if can't explain it right now in
>>> details.
>>>
>>> Thanks
>>> Edgar
>>>
>>> Jeff Squyres wrote:
>>>> On Dec 2, 2008, at 11:27 AM, Edgar Gabriel wrote:
>>>>> so I ran a couple of tests today and I can not confirm your
>>>>> statement. I wrote simple a simple test code where a process
>>>>> first sets an affinity mask and than spawns a number of threads.
>>>>> The threads modify the affinity mask and every thread
>>>>> ( including the master thread) print out there affinity mask at
>>>>> the end.
>>>>>
>>>>> With sched_getaffinity() and sched_setaffinity() it was indeed
>>>>> such that the master thread had the same affinity mask as the
>>>>> thread that it spawned. This means, that the modification of the
>>>>> affinity mask by a new thread in fact did affect the master
>>>>> thread.
>>>>>
>>>>> Executing the same codesquence however using the libnuma calls,
>>>>> the master thread however was not affected by the new affinity
>>>>> mask of the children. So clearly, libnuma must be doing
>>>>> something differently.
>>>> What distro/version of Linux are you using, and what version of
>>>> libnuma?
>>>> Libnuma v2.0.x very definitely is just a wrapper around the
>>>> syscall for sched_setaffinity(). I downloaded it from:
>>>> ftp://oss.sgi.com/www/projects/libnuma/download
>>>
>>> --
>>> Edgar Gabriel
>>> Assistant Professor
>>> Parallel Software Technologies Lab http://pstl.cs.uh.edu
>>> Department of Computer Science University of Houston
>>> Philip G. Hoffman Hall, Room 524 Houston, TX-77204, USA
>>> Tel: +1 (713) 743-3857 Fax: +1 (713) 743-3335
>>> _______________________________________________
>>> users mailing list
>>> users_at_[hidden]
>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>
> --
> Edgar Gabriel
> Assistant Professor
> Parallel Software Technologies Lab http://pstl.cs.uh.edu
> Department of Computer Science University of Houston
> Philip G. Hoffman Hall, Room 524 Houston, TX-77204, USA
> Tel: +1 (713) 743-3857 Fax: +1 (713) 743-3335
> _______________________________________________
> users mailing list
> users_at_[hidden]
> http://www.open-mpi.org/mailman/listinfo.cgi/users

-- 
Jeff Squyres
Cisco Systems