Open MPI logo

Hardware Locality Development Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Hardware Locality Development mailing list

Subject: Re: [hwloc-devel] hwloc_distances as utility?
From: Brice Goglin (Brice.Goglin_at_[hidden])
Date: 2011-07-05 16:28:36

Le 05/07/2011 22:04, Jiri Hladky a écrit :
> Well, this is interesting. numactl --hardware shows the number of
> hops, regarding to the information from that private BZ.

I think this is wrong. numactl takes everything from sysfs as far as I
can tell. On x86, sysfs distances are ACPI SLIT latencies (memory
latencies that are normalized to 10 for latencies from one cpu to its
local memory). A couple months ago, I checked all Linux ports that show
distances in sysfs. All of them report memory latencies, except the SGI
IP27 as mentioned previously (this one indeed shows number of hops (0
when local) and it makes a lot of sense for this architecture).

That said, the latency that is reported in sysfs is definitely related
to the number of hops, and I assume that BIOS vendors look at the number
of hops when deciding what to put in the ACPI SLIT distances. It's not
something measured at boot. It's kind of hardcoded by/in the BIOS and
decided as the same time the routing is computed. But still the ACPI
spec says that distances are normalized latencies.

One problem I see with the number of hops is that it doesn't make sense
on some machines. On some 8-socket AMD machines (such as
8amd64-4n2c.tar.bz2 below) , the hypertransport route between some
sockets varies with the type of packet (response or request) and the
direction. So the number of hops ends up being asymmetric, depends on
read/write, and can be half of an integer.

> could me send me the output of hwloc-gather-topology for some NUMA
> box? I don't have access to any NUMA running a recent version of kernel.

Look at tests/linux/ in the hwloc SVN. The following tarballs contain
NUMA architectures. Some of these were gathered while running old
kernels, but I don't think it matters because Linux/sysfs reports what
the BIOS without changing much of it.

8amd64-4n2c.tar.bz2 (2 hops, but distance is reported as local(10) or
remote (20))
16amd64-8n2c.tar.bz2 (up to 3 hops, but local or remote too)
48amd64-4d2n6c-sparse.tar.bz2 (2 hops IIRC, local(10), remote(16) or
"very remote"(22))
96em64t-4n4d3ca2co.tar.bz2 (1 hop, local(10) or remote(22))
16ia64-8n2s.tar.bz2 (2 hops, 10/25/29)
256ia64-64n2s2c.tar.bz2 (big SGI Altix machine with several hops,