ok, so I digged a little deeper, and have some good news. Let me start
with a set of routines, that we didn't even discuss yet, but which works
for setting thread affinity, and discuss then libnuma and
On linux systems, the pthread library has a set of routines to modify
and determine thread-affinity related information:
int pthread_setaffinity_np (pthread_t __th, size_t __cpusetsize,
const cpu_set_t *__cpuset);
int pthread_getaffinity_np (pthread_t __th, size_t __cpusetsize,
These two routines can be used to modify the affinity of an existing
thread. If you would like to modify the affinity of a thread *before*
creating it, you can use a similar routines.
int pthread_attr_setaffinity_np (pthread_attr_t *__attr,
const cpu_set_t *__cpuset)
I tested the first two routines, and they did work for me.
Now to libnuma vs. sched_setaffinity: after digging a little deeper in
the libnuma sources, I realized that one of the differences on what they
do vs. what I did in my testcases was, that libnuma uses the
sched_setaffinity() calls with a pid of 0, instead of determining the
pid using the getpid() function. According to the sched_setaffinity()
manpages, pid of zero means 'apply the new rules to the current
process', but it does in fact mean 'to the current task/thread'. I wrote
a set of tests, where I used sched_setaffinity() with the pid zero, and
I was in fact able to modify the affinity of an individual thread using
sched_setaffinity(). If you pass in the pid of the process, it will
affect the affinity of all threads of that process.
Bottom line is, you can modify the affinity of a thread using both
libnuma on a per socket basis and the sched_setaffinity() calls on a per
core basis. Alternatively, you can use the pthread_setaffinity_np()
function to modify the affinity of a thread using a cpu_set_t similar to
Jeff Squyres wrote:
> Fair enough; let me know what you find. It would be good to understand
> exactly why you're seeing what you're seeing...
> On Dec 2, 2008, at 5:47 PM, Edgar Gabriel wrote:
>> its on OpenSuSE 11 with kernel 184.108.40.206. I don't know the libnuma
>> library version, but I suspect that its fairly new.
>> I will try to investigate that in the next days a little more. I do
>> think that they use sched_setaffinity() underneath the hood (because
>> in one of my failed attempts when I passed in the wrong argument, I
>> got actually the same error message that I got earlier with
>> sched_setaffinity), but they must do something additionally underneath.
>> Anyway, I just wanted to report the result, and that there is
>> obviously a difference, even if can't explain it right now in details.
>> Jeff Squyres wrote:
>>> On Dec 2, 2008, at 11:27 AM, Edgar Gabriel wrote:
>>>> so I ran a couple of tests today and I can not confirm your
>>>> statement. I wrote simple a simple test code where a process first
>>>> sets an affinity mask and than spawns a number of threads. The
>>>> threads modify the affinity mask and every thread ( including the
>>>> master thread) print out there affinity mask at the end.
>>>> With sched_getaffinity() and sched_setaffinity() it was indeed such
>>>> that the master thread had the same affinity mask as the thread that
>>>> it spawned. This means, that the modification of the affinity mask
>>>> by a new thread in fact did affect the master thread.
>>>> Executing the same codesquence however using the libnuma calls, the
>>>> master thread however was not affected by the new affinity mask of
>>>> the children. So clearly, libnuma must be doing something differently.
>>> What distro/version of Linux are you using, and what version of libnuma?
>>> Libnuma v2.0.x very definitely is just a wrapper around the syscall
>>> for sched_setaffinity(). I downloaded it from:
>> Edgar Gabriel
>> Assistant Professor
>> Parallel Software Technologies Lab http://pstl.cs.uh.edu
>> Department of Computer Science University of Houston
>> Philip G. Hoffman Hall, Room 524 Houston, TX-77204, USA
>> Tel: +1 (713) 743-3857 Fax: +1 (713) 743-3335
>> users mailing list
Parallel Software Technologies Lab http://pstl.cs.uh.edu
Department of Computer Science University of Houston
Philip G. Hoffman Hall, Room 524 Houston, TX-77204, USA
Tel: +1 (713) 743-3857 Fax: +1 (713) 743-3335