Open MPI logo

Hardware Locality Development Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Hardware Locality Development mailing list

Subject: Re: [hwloc-devel] SWIG bindings
From: Brice Goglin (Brice.Goglin_at_[hidden])
Date: 2010-12-03 03:54:29


Le 02/12/2010 22:25, Bernd Kallies a écrit :
>
>> Do you have any feel for if there are particular bottlenecks in hwloc / lstopo that make it take so long? I wonder if we should just attack those (if possible)...? Samuel and Brice have done all the work in the guts of the API, so they might know offhand if there are places that can be optimized or not...
>>
> Hmm. I did no profiling. The machines in question have 64 NUMA nodes
> with 16 logical CPUs, each. The topology depth is 10. So parsing
> of /sys/devices/system/node/* and evaluating the distance matrix to
> fiddle out the topology tree should be quite expensive. But I guess this
> statement is trivial and does not help very much.
>

We should really encourage people to use XML in such cases. Setting
HWLOC_XMLFILE=/path/to/exported/file.xml in the environment should just
work (as long as you update the XML file major hwloc releases or os).
Maybe we should add a dedicated section about this in the documentation?
Something like "Speeding up hwloc on large nodes"? And maybe even
encourage distro-packager to create a XML export file under /var/lib
with an advice to add HWLOC_XMLFILE to /etc/environment if they care
about hwloc/HPC?

Anyway Bernd, can you export a XML on this nice machine and reload it
and see how long it takes? I hope all the bottlenecks are in the Linux
backend parsing /sys and /proc, not in the actual hwloc core.

By the way, we're not the only project with little scalability problems
on very large machines: https://lkml.org/lkml/2010/12/3/19 :)

Brice