Open MPI logo

Hardware Locality Development Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Hardware Locality Development mailing list

Subject: [hwloc-devel] HWLOC_FSROOT in libnuma
From: Brice Goglin (Brice.Goglin_at_[hidden])
Date: 2012-02-21 00:50:19

While debugging libnuma helpers, I needed a way to emulate remote
topologies in libnuma as hwloc does so that we can do
  HWLOC_FSROOT=/foo/bar tests/linux-libnuma
on any of my existing remote topologies. I modified 2.0.8-rc3 with the
attached patch so that it honors HWLOC_FSROOT when reading /sys and
/proc files.

It's not perfect:
* We need to retrieve the distant /proc/self/status and manually add it
to $FSROOT/proc/self/ (libnuma needs it, hwloc doesn't gather it)
* The remote topology cannot use cpumaps that are larger than the local
kernel cpumap. Basically, you need to remove starting zeros in
$FSROOT/sys/devices/system/node/node*/cpumap until it's not longer than
your local /sys/devices/system/node/node0/cpumap
* Some corner cases don't case such as highly sparse node ids don't
work, maybe because CONFIG_NODES_SHIFT=6 in my kernel

The patch also brings back the old "numa_all_nodes" behavior (before
libnuma 2.0.6) wrt nodes with no memory if HWLOC_OLD_LIBNUMA=1. But this
shouldn't matter anyway because I removed nodemask_t helpers from trunk.
Still no reply to my bug report on numa-devel about this

I am sharing this so that it doesn't get lost, in case somebody ever has
to debug this again.