Open MPI logo

Hardware Locality Development Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Hardware Locality Development mailing list

Subject: Re: [hwloc-devel] hwloc_distances as utility?
From: Jiri Hladky (hladky.jiri_at_[hidden])
Date: 2011-07-14 17:49:02


Hi Brice,

> Well, this is interesting. numactl --hardware shows the number of
> > hops, regarding to the information from that private BZ.
>
> I think this is wrong. numactl takes everything from sysfs as far as I
> can tell. On x86, sysfs distances are ACPI SLIT latencies (memory
> latencies that are normalized to 10 for latencies from one cpu to its
> local memory). A couple months ago, I checked all Linux ports that show
> distances in sysfs. All of them report memory latencies, except the SGI
> IP27 as mentioned previously (this one indeed shows number of hops (0
> when local) and it makes a lot of sense for this architecture).
>

Yes, you are absolutly right. Sorry about it. It's better to check some
sources than to rely on my own memory:-)

> One problem I see with the number of hops is that it doesn't make sense
> on some machines. On some 8-socket AMD machines (such as
> 8amd64-4n2c.tar.bz2 below) , the hypertransport route between some
> sockets varies with the type of packet (response or request) and the
> direction. So the number of hops ends up being asymmetric, depends on
> read/write, and can be half of an integer.
>

Yes, this is exactly what I have seen as well. This even true for 4 socket
AMD Magny Cours system.

> Look at tests/linux/ in the hwloc SVN. The following tarballs contain
> NUMA architectures. Some of these were gathered while running old
> kernels, but I don't think it matters because Linux/sysfs reports what
> the BIOS without changing much of it.
>

I will. The thing is that I'm pretty bu\sy right now, I need to finish a
project before going on vacation. I hope to look into in September.

Regards
Jirka