Open MPI logo

Hardware Locality Development Mailing List Archives

  |   Home   |   Support   |   FAQ   |  

This web mail archive is frozen.

This page is part of a frozen web archive of this mailing list.

You can still navigate around this archive, but know that no new mails have been added to it since July of 2016.

Click here to be taken to the new web archives of this list; it includes all the mails that are in this frozen archive plus all new mails that have been sent to the list since it was migrated to the new archives.

Subject: Re: [hwloc-devel] multiple simultaneous topology inits?
From: Jeff Squyres (jsquyres) (jsquyres_at_[hidden])
Date: 2011-03-01 06:34:59


FWIW OMPI is moving to a model like that.

Sent from my phone. No type good.

On Mar 1, 2011, at 4:47 AM, "Bernd Kallies" <kallies_at_[hidden]> wrote:

> On Mon, 2011-02-28 at 22:17 +0100, Brice Goglin wrote:
>> Le 28/02/2011 22:04, Jeff Squyres a écrit :
>>> That being said, someone cited on this list a long time ago that running the hwloc detection on very large machines (e.g., SGI machines with 1000+ cores) takes on the order of seconds (because it traverses /sys, etc.). So if you want your tool to be used on machines like that, then it might be better to do the discovery once and share that data among your threads.
>>>
>>
>> People running on such large machines should really export the machine
>> topology to XML once and reload from there all the time.
>
> Btw. lstopo on such a large machine (64 NUMA nodes, 1024 logical CPUs)
> takes about 0.6 seconds at our site.
>
> This is accepteable for scripts, that run only frequently. It is also
> accepteable for executables that need the topology info at start time
> (e.g. pbs_mom of Torque).
>
> To calculate topology-based pinning schemes and do process pinning (like
> done e.g. by OpenMPI or MVAPICH2) this is too long, when every process
> (MPI task) or thread loads the topology in parallel. But exporting an
> XML topology and using this for this purpose is inaccepteable, when
> Linux cpusets are used, because one needs the topology of a subset of
> the machine depending on the caller context. What we currently do is to
> let only one process per machine load the topology, and distribute the
> essentials needed for pinning to the remaining processes.
>
> BK
>
>> Brice
>>
>>
>>
>> _______________________________________________
>> hwloc-devel mailing list
>> hwloc-devel_at_[hidden]
>> http://www.open-mpi.org/mailman/listinfo.cgi/hwloc-devel
>
> --
> Dr. Bernd Kallies
> Konrad-Zuse-Zentrum für Informationstechnik Berlin
> Takustr. 7
> 14195 Berlin
> Tel: +49-30-84185-270
> Fax: +49-30-84185-311
> e-mail: kallies_at_[hidden]
>
> _______________________________________________
> hwloc-devel mailing list
> hwloc-devel_at_[hidden]
> http://www.open-mpi.org/mailman/listinfo.cgi/hwloc-devel