Open MPI logo

Hardware Locality Development Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Hardware Locality Development mailing list

Subject: Re: [hwloc-devel] roadmap
From: Samuel Thibault (samuel.thibault_at_[hidden])
Date: 2010-09-22 10:50:14

Please people on the list tell us what your needs are, else we can't for
sure design things a proper way!

Brice Goglin, le Wed 22 Sep 2010 14:13:20 +0200, a écrit :
> Le 22/09/2010 13:36, Jeff Squyres a écrit :
> > On Sep 22, 2010, at 4:38 AM, Brice Goglin wrote:
> >> There are still some problems to solve in the membind branch:
> >> * Some OS bind the process too when you bind memory. I see the following
> >> solutions:
> >> + Add a flag such as HWLOC_MEMBIND_EVEN_IF_FAR_FROM_PROCESS so that
> >> the user can explicitly refuse memory binding if it may break process
> >> binding
> >> + Drop hwloc_set_membind on these OSes and add a
> >> hwloc_set_cpumembind() to bind both
> >> + Make both process and memory binding do nothing if the STRICT flag
> >> is given. But I'd rather not play too much with this flag.
> >> + Drop support for memory binding on these OS.
> >> + Drop these OS.
> >>
> > What OS's are you specifically referring to?
> IIRC, it was AIX and Solaris.

topology-aix.c:/* TODO: set_membind set_proc_membind, document that it binds threads too. */
topology-solaris.c:/* TODO: set_membind set_proc_membind thanks to lgrp_affinity, document that it binds threads too. */

> > How about adding a query function that says what will happen for hwloc_set_membind()
> I like it, we can put this in the output of hwloc_topology_get_support.

A concern is that most people will probably not bother checking that.
Note that in some cases cpubind could do the membinding too. For
instance, the extension we have added to the Solaris port to use lgrp
for cpubind will actually also bind memory.

Also, it would be helpful if the function to allocate memory on some
given node was allowed to change the default allocation policy, because
in such case we'd be able to say that it works on all OSes. For now it

A typical scenario I see is:

- allocate memory in the main thread, explicitly directed to some
- start threads, bind them.


- allocate memory in the main thread, with a first-touch policy.
- start threads, bind them, let them touch data.


- start threads, bind them (and their memory), let them allocate data.

set_cpumembind() here should be fine for all cases. One case that
wouldn't work is for instance

- main thread binds itself, starts some other threads, which bind
  themselves somewhere else.
- while other threads are busy, main thread allocates some data for
  other threads, and thus needs to change the allocation policy, which
  may unbind it.

But in such case (for which the OS doesn't provide any facility to
address it better anyway), one could assume that the OS will just
schedule the main thread where it is alone, and achieve directed
allocation. Re-binding the thread might however be a good thing to do
after doing the memory allocation, to be better confident with the cpu

Another point that comes to mind is whether people really want to
control both where a thread executes and where it allocates memory _at
the same time_. Letting cpubind change the memory allocation policy and
letting membind change the cpu scheduling policy is fine, provided the
user knows that one might influence the other, and thus if he calls
set_cpubind and set_membind with different cpu sets, after calling one
for some reason, he'll have to call the other one again.

I think that's where we can't provide portability by just providing
functions, but we also need to provide methodology documentation.

> I wonder if there are some other cases where the STRICT flag could be
> dropped in favor of such an informative stuff.

In all cases I know, STRICT is not to be dropped because it indeed
selects between strict binding (never execute somwhere else, at worse
fail) or not.

> > Just curious -- on these OS's, what happens if you:
> >
> > - bind proc to A
> > - bind memory to B (which then also re-binds proc to B)
> > - re-bind proc to A
> >
> > Is the memory binding then lost?
> I'll let Samuel comment on this.

That would depend on the OS of course :) Data migration is actually
quite rare in OSes, it's often really implemented only for machine

For process binding, I believe OSes will tend to do the data migration.
For thread binding, I doubt any OS would do the data migration, as data
aren't supposed to be attached to any particular thread of the process.

> >> * cpuset and nodeset structures are the same, they are both manipulated
> >> with hwloc_cpuset_foo functions. So maybe rename into hwloc_set_t and
> >> hwloc_set_foo functions. With #define and aliases to not break API/ABIs.
> >>
> > I'm in favor of this -- it would end the overloading of the term "cpuset" between hwloc and cpuset.
> hwloc_set_*? hwloc_objset* ? Anything better?

objset would be confusing with hwloc objects.


> hwloc_set_* might not be the best since we would have a hwloc_set_set()
> function to set one bit :)


> By the way, hwloc_cpuset_cpu() and hwloc_cpuset_all_but_cpu() should be
> renamed too. hwloc_set_onlyone() and hwloc_set_allbutone() maybe?

For instance, I'd let a native english speaker decide, however :)