Open MPI logo

Hardware Locality Development Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Hardware Locality Development mailing list

Subject: [hwloc-devel] hwloc powerpc rhel5 and power7 patch
From: Alexey Kardashevskiy (aik_at_[hidden])
Date: 2010-09-16 00:10:08


Hi!

There are 2 problems with the current HWLOC code. The questions are at
the bottom.

1. Old kernels (RHEL5.*) do expose some numa nodes via sysfs but there
is no information regarting cache (L1/L2/L3) and CPU threads. RHEL6 does
that. The proposed patch parses PowerPC's /proc/device-tree and add
necessary information into the topology.

2. The HWLOC expects numa nodes to be numbered consecutively, like
1-2-3-4-5.... However this is not necessary true for PowerPC with LPARs
or on systems with numa hotswap (do they exist? don't know). This was
before the patch:

=========================
os node 0 has cpuset 0xffffffff
os node 1 has cpuset 0xffffffff,0x0
os node 4 has cpuset 0xffffffff,,0x0
os node 5 has cpuset 0xffffffff,,,0x0
os node 8 has cpuset 0xffffffff,,,,0x0
os node 9 has cpuset 0xffffffff,,,,,0x0
os node 12 has cpuset 0xffffffff,,,,,,0x0
os node 13 has cpuset 0xffffffff,,,,,,,0x0
node distance matrix:
       0 1 2 3 4 5 6 7 8 9 10 11 12 13
   0 10 20 40 40 40 40 40 40 0 1 128 3596701896 0 1
   1 20 10 40 40 40 40 40 40 4095 3642405872 4095 3642406288 0
65536
   2 128 3596490848 4095 3642406160 4095 3642406048 0 0 128
3597792932 0 0 0 0
   3 128 3598856792 0 0 0 0 0 1 0 218840 0 1 0 0
   4 40 40 10 20 40 40 40 40 128 3596902928 128 3596700232 4095
3642406320
   5 40 40 20 10 40 40 40 40 0 5 4095 3642406432 0 0
   6 4095 3642406256 0 0 128 3596923832 256 276108416 4095
3642406272 0 0 0 0
   7 0 0 0 0 256 276173984 128 3598846040 0 191376 128
3598846016 256 276108400
   8 40 40 40 40 10 20 40 40 4095 2587230208 4095 2587260160 0 0
   9 40 40 40 40 20 10 40 40 4095 3642406320 4095 3642406464 0 0
  10 0 0 0 0 0 0 0 0 0 0 0 0 0 0
  11 0 0 0 0 0 0 0 0 128 3596680064 128 3596687552 128
3596679872
  12 40 40 40 40 40 40 10 20 128 3597793376 128 3598856792 128
3597315568
  13 40 40 40 40 40 40 20 10 0 0 128 3598779600 429496729
2576980377
distance matrix asymmetric ([0,2]=40 != [2,0]=128), aborting
=========================

This is how it is supposed to look like:
=========================
node distance matrix:
       0 1 4 5 8 9 12 13
   0 10 20 40 40 40 40 40 40
   1 20 10 40 40 40 40 40 40
   4 40 40 10 20 40 40 40 40
   5 40 40 20 10 40 40 40 40
   8 40 40 40 40 10 20 40 40
   9 40 40 40 40 20 10 40 40
  12 40 40 40 40 40 40 10 20
  13 40 40 40 40 40 40 20 10
trying to group NUMANode objects into misc objects according to physical
distances
found minimal distance 20 between objects
object 1 is minimally connected to 0
found transitive graph with 2 objects with minimal distance 20
object 3 is minimally connected to 2
found transitive graph with 2 objects with minimal distance 20
object 5 is minimally connected to 4
found transitive graph with 2 objects with minimal distance 20
object 7 is minimally connected to 6
found transitive graph with 2 objects with minimal distance 20
adding misc object with 2 objects and cpuset 0xffffffff,0xffffffff
adding misc object with 2 objects and cpuset 0xffffffff,0xffffffff,,0x0
adding misc object with 2 objects and cpuset 0xffffffff,0xffffffff,,,,0x0
adding misc object with 2 objects and cpuset 0xffffffff,0xffffffff,,,,,,0x0
group distances:
15 40 40 40
40 15 40 40
40 40 15 40
40 40 40 15
trying to group Group objects into misc objects according to physical
distances
found minimal distance 40 between objects
object 1 is minimally connected to 0
object 2 is minimally connected to 0
object 3 is minimally connected to 0
found transitive graph with 4 objects with minimal distance 40
=========================

The questions are:
1. What should I change in my patch to have it committed into the svn?
Specifically:
- where do I put IBM-specific code?
- may be there is a better way to detect that no cache info was fetched
from sysfs
- is the coding style ok? :-)

2. Do not I miss something in my patch in order to solve the problems
mentioned in the beginning of this mail?

Thank you.