Open MPI logo

Hardware Locality Development Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Hardware Locality Development mailing list

Subject: Re: [hwloc-devel] More memory binding questions
From: Samuel Thibault (samuel.thibault_at_[hidden])
Date: 2011-01-05 05:43:52

Jeff Squyres, le Tue 04 Jan 2011 22:22:41 +0100, a écrit :
> hwloc_alloc_membind
> hwloc_alloc_membind_nodeset
> What does "binding cannot be enforced" mean (i.e., when EXDEV is returned)?

It's explained in more details in the header of the documentation

“errno is set to EXDEV when the requested cpuset can not be enforced
(e.g. some systems only allow one CPU, and some other systems only allow
one NUMA node).”

I.e. it's the given combination of cpus or nodes which the OS doesn't
support, typically because the combination crosses some device boundary,
thus the choice of EXDEV.

> Assumedly the system supports binding (otherwise ENOSYS would have been returned). Does it mean, for example, that I asked for X memory on node Y, but node Y didn't have X memory available on node Y?

That case would rather be OS-dependent.

> When EXDEV is returned, is valid memory returned (that must be freed via hwloc_free), or does the entire allocation fail?

It completely fails. But as documented in the header of the
documentation section, that only happens when STRICT is passed.

> hwloc_get_area_membind
> hwloc_get_area_membind_nodeset
> Is the purpose of these 2 functions to query the memory binding policy for an (addr, len) tuple in the current process?


> If so, what happens if there are multiple memory binding policies in effect for that tuple?

It seems I forgot to copy/paste the documentation from the CPU part
about the STRICT flag, I've commited it:

“When retrieving the binding of a memory range, this flag checks
whether all its pages actually have the same binding. If the flag is
not given, the binding of each page will be accumulated.”

> What is the purpose of the IN flags argument?

Answered above :) At least the STRICT flag.

> Taking a guess: can you pass _PROCESS or _THREAD in to query the binding policy for this specific thread or the entire process? (and all other flags are ignored or illegal?)

It does not make sense in the case of area_get_membind, since memory
binding for an area is related to the area, not to who accesses it (even
in the firsttouch case, where the policy is firsttouch).

> hwloc_get_membind
> hwloc_get_membind_nodeset
> Is the purpose of these 2 functions to query the memory binding polocy for the entire current process (or a thread in the current process)?

Ah, that part of the cpubind documentation didn't get copied
either. I have commited it:

specified, the current process is assumed to be mono-thread. This is the
most portable way as it permits hwloc to use either process-based OS
functions or thread-based OS functions, whichever are available.”

> hwloc_get_proc_membind
> hwloc_get_proc_membind_nodeset
> It took me a while to notice the PID argument in this function, and that this was the main difference from hwloc_get_membind*. I assume that the documentation statement "Get current process memory binding in cpuset cpuset." is actually incorrect -- right?

Oops, indeed. That's the problem with painfully documenting a big series
of very similar functions...

> It shouldn't say "current" -- it should indicate that you can query the memory binding policy for the process specified by the pid argument, right? If so, is it equivalent to calling hwloc_get_membind* in that process?


> Assumedly, you need some kind of rights on the target process -- either you're the owner of the process, or are superuser, ...or have some other administrative rights on the target. Right?