Open MPI logo

Hardware Locality Development Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Hardware Locality Development mailing list

Subject: Re: [hwloc-devel] multiple simultaneous topology inits?
From: Bernd Kallies (kallies_at_[hidden])
Date: 2011-03-01 05:16:43

On Tue, 2011-03-01 at 11:00 +0100, Brice Goglin wrote:
> Le 01/03/2011 10:46, Bernd Kallies a écrit :
> > To calculate topology-based pinning schemes and do process pinning (like
> > done e.g. by OpenMPI or MVAPICH2) this is too long, when every process
> > (MPI task) or thread loads the topology in parallel. But exporting an
> > XML topology and using this for this purpose is inaccepteable, when
> > Linux cpusets are used, because one needs the topology of a subset of
> > the machine depending on the caller context. What we currently do is to
> > let only one process per machine load the topology, and distribute the
> > essentials needed for pinning to the remaining processes.
> >
> You can still export one XML file per cpuset and load the right one in
> each process. If the batch scheduler creates the cpuset, my feeling is
> that the batch scheduler should create the XML file at the same time,
> and do something like export HWLOC_XMLFILE=mycpuset.xml when launching
> processes.

Yes, that's a good idea, as long as the extent of the cpuset is not
modified after the batch scheduler created it (e.g. because it is not
aware of SMT, or because the scheduler does not handle node access
policies correctly). Things are in progress regarding this for Torque,
so letting it create a HWLOC_XMLFILE is definitely worth to implement.

> Also, in 1.2, we'll have a hwloc_topology_restrict() function which will
> let you load the whole machine topology and then restrict it to whatever
> part of it (a part is defined by a hwloc_cpuset_t). We'll need to make
> sure that you'll have everything you need to get your cpuset's
> hwloc_cpuset_t.

Thanks for the info. Such a thing is definitely needed.


> Brice
> _______________________________________________
> hwloc-devel mailing list
> hwloc-devel_at_[hidden]

Dr. Bernd Kallies
Konrad-Zuse-Zentrum für Informationstechnik Berlin
Takustr. 7
14195 Berlin
Tel: +49-30-84185-270
Fax: +49-30-84185-311
e-mail: kallies_at_[hidden]