Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: Re: [OMPI users] Hybrid program
From: Edgar Gabriel (gabriel_at_[hidden])
Date: 2008-12-05 10:45:23


ok, so I digged a little deeper, and have some good news. Let me start
with a set of routines, that we didn't even discuss yet, but which works
for setting thread affinity, and discuss then libnuma and
sched_setaffinity() again.

-----------
On linux systems, the pthread library has a set of routines to modify
and determine thread-affinity related information:

#define __USE_GNU

int pthread_setaffinity_np (pthread_t __th, size_t __cpusetsize,
                             const cpu_set_t *__cpuset);
int pthread_getaffinity_np (pthread_t __th, size_t __cpusetsize,
                             cpu_set_t *__cpuset)

These two routines can be used to modify the affinity of an existing
thread. If you would like to modify the affinity of a thread *before*
creating it, you can use a similar routines.

int pthread_attr_setaffinity_np (pthread_attr_t *__attr,
                                  size_t __cpusetsize,
                                  const cpu_set_t *__cpuset)

I tested the first two routines, and they did work for me.
-----------

Now to libnuma vs. sched_setaffinity: after digging a little deeper in
the libnuma sources, I realized that one of the differences on what they
do vs. what I did in my testcases was, that libnuma uses the
sched_setaffinity() calls with a pid of 0, instead of determining the
pid using the getpid() function. According to the sched_setaffinity()
manpages, pid of zero means 'apply the new rules to the current
process', but it does in fact mean 'to the current task/thread'. I wrote
a set of tests, where I used sched_setaffinity() with the pid zero, and
I was in fact able to modify the affinity of an individual thread using
sched_setaffinity(). If you pass in the pid of the process, it will
affect the affinity of all threads of that process.

Bottom line is, you can modify the affinity of a thread using both
libnuma on a per socket basis and the sched_setaffinity() calls on a per
core basis. Alternatively, you can use the pthread_setaffinity_np()
function to modify the affinity of a thread using a cpu_set_t similar to
sched_setaffinity.

Thanks
Edgar

Jeff Squyres wrote:
> Fair enough; let me know what you find. It would be good to understand
> exactly why you're seeing what you're seeing...
>
>
> On Dec 2, 2008, at 5:47 PM, Edgar Gabriel wrote:
>
>> its on OpenSuSE 11 with kernel 2.6.25.11. I don't know the libnuma
>> library version, but I suspect that its fairly new.
>>
>> I will try to investigate that in the next days a little more. I do
>> think that they use sched_setaffinity() underneath the hood (because
>> in one of my failed attempts when I passed in the wrong argument, I
>> got actually the same error message that I got earlier with
>> sched_setaffinity), but they must do something additionally underneath.
>>
>> Anyway, I just wanted to report the result, and that there is
>> obviously a difference, even if can't explain it right now in details.
>>
>> Thanks
>> Edgar
>>
>> Jeff Squyres wrote:
>>> On Dec 2, 2008, at 11:27 AM, Edgar Gabriel wrote:
>>>> so I ran a couple of tests today and I can not confirm your
>>>> statement. I wrote simple a simple test code where a process first
>>>> sets an affinity mask and than spawns a number of threads. The
>>>> threads modify the affinity mask and every thread ( including the
>>>> master thread) print out there affinity mask at the end.
>>>>
>>>> With sched_getaffinity() and sched_setaffinity() it was indeed such
>>>> that the master thread had the same affinity mask as the thread that
>>>> it spawned. This means, that the modification of the affinity mask
>>>> by a new thread in fact did affect the master thread.
>>>>
>>>> Executing the same codesquence however using the libnuma calls, the
>>>> master thread however was not affected by the new affinity mask of
>>>> the children. So clearly, libnuma must be doing something differently.
>>> What distro/version of Linux are you using, and what version of libnuma?
>>> Libnuma v2.0.x very definitely is just a wrapper around the syscall
>>> for sched_setaffinity(). I downloaded it from:
>>> ftp://oss.sgi.com/www/projects/libnuma/download
>>
>> --
>> Edgar Gabriel
>> Assistant Professor
>> Parallel Software Technologies Lab http://pstl.cs.uh.edu
>> Department of Computer Science University of Houston
>> Philip G. Hoffman Hall, Room 524 Houston, TX-77204, USA
>> Tel: +1 (713) 743-3857 Fax: +1 (713) 743-3335
>> _______________________________________________
>> users mailing list
>> users_at_[hidden]
>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>
>

-- 
Edgar Gabriel
Assistant Professor
Parallel Software Technologies Lab      http://pstl.cs.uh.edu
Department of Computer Science          University of Houston
Philip G. Hoffman Hall, Room 524        Houston, TX-77204, USA
Tel: +1 (713) 743-3857                  Fax: +1 (713) 743-3335