Open MPI logo

Hardware Locality Development Mailing List Archives

  |   Home   |   Support   |   FAQ   |  

This web mail archive is frozen.

This page is part of a frozen web archive of this mailing list.

You can still navigate around this archive, but know that no new mails have been added to it since July of 2016.

Click here to be taken to the new web archives of this list; it includes all the mails that are in this frozen archive plus all new mails that have been sent to the list since it was migrated to the new archives.

Subject: [hwloc-devel] hwloc powerpc rhel5 and power7 patch
From: Alexey Kardashevskiy (aik_at_[hidden])
Date: 2010-09-16 00:10:08


Hi!

There are 2 problems with the current HWLOC code. The questions are at
the bottom.

1. Old kernels (RHEL5.*) do expose some numa nodes via sysfs but there
is no information regarting cache (L1/L2/L3) and CPU threads. RHEL6 does
that. The proposed patch parses PowerPC's /proc/device-tree and add
necessary information into the topology.

2. The HWLOC expects numa nodes to be numbered consecutively, like
1-2-3-4-5.... However this is not necessary true for PowerPC with LPARs
or on systems with numa hotswap (do they exist? don't know). This was
before the patch:

=========================
os node 0 has cpuset 0xffffffff
os node 1 has cpuset 0xffffffff,0x0
os node 4 has cpuset 0xffffffff,,0x0
os node 5 has cpuset 0xffffffff,,,0x0
os node 8 has cpuset 0xffffffff,,,,0x0
os node 9 has cpuset 0xffffffff,,,,,0x0
os node 12 has cpuset 0xffffffff,,,,,,0x0
os node 13 has cpuset 0xffffffff,,,,,,,0x0
node distance matrix:
       0 1 2 3 4 5 6 7 8 9 10 11 12 13
   0 10 20 40 40 40 40 40 40 0 1 128 3596701896 0 1
   1 20 10 40 40 40 40 40 40 4095 3642405872 4095 3642406288 0
65536
   2 128 3596490848 4095 3642406160 4095 3642406048 0 0 128
3597792932 0 0 0 0
   3 128 3598856792 0 0 0 0 0 1 0 218840 0 1 0 0
   4 40 40 10 20 40 40 40 40 128 3596902928 128 3596700232 4095
3642406320
   5 40 40 20 10 40 40 40 40 0 5 4095 3642406432 0 0
   6 4095 3642406256 0 0 128 3596923832 256 276108416 4095
3642406272 0 0 0 0
   7 0 0 0 0 256 276173984 128 3598846040 0 191376 128
3598846016 256 276108400
   8 40 40 40 40 10 20 40 40 4095 2587230208 4095 2587260160 0 0
   9 40 40 40 40 20 10 40 40 4095 3642406320 4095 3642406464 0 0
  10 0 0 0 0 0 0 0 0 0 0 0 0 0 0
  11 0 0 0 0 0 0 0 0 128 3596680064 128 3596687552 128
3596679872
  12 40 40 40 40 40 40 10 20 128 3597793376 128 3598856792 128
3597315568
  13 40 40 40 40 40 40 20 10 0 0 128 3598779600 429496729
2576980377
distance matrix asymmetric ([0,2]=40 != [2,0]=128), aborting
=========================

This is how it is supposed to look like:
=========================
node distance matrix:
       0 1 4 5 8 9 12 13
   0 10 20 40 40 40 40 40 40
   1 20 10 40 40 40 40 40 40
   4 40 40 10 20 40 40 40 40
   5 40 40 20 10 40 40 40 40
   8 40 40 40 40 10 20 40 40
   9 40 40 40 40 20 10 40 40
  12 40 40 40 40 40 40 10 20
  13 40 40 40 40 40 40 20 10
trying to group NUMANode objects into misc objects according to physical
distances
found minimal distance 20 between objects
object 1 is minimally connected to 0
found transitive graph with 2 objects with minimal distance 20
object 3 is minimally connected to 2
found transitive graph with 2 objects with minimal distance 20
object 5 is minimally connected to 4
found transitive graph with 2 objects with minimal distance 20
object 7 is minimally connected to 6
found transitive graph with 2 objects with minimal distance 20
adding misc object with 2 objects and cpuset 0xffffffff,0xffffffff
adding misc object with 2 objects and cpuset 0xffffffff,0xffffffff,,0x0
adding misc object with 2 objects and cpuset 0xffffffff,0xffffffff,,,,0x0
adding misc object with 2 objects and cpuset 0xffffffff,0xffffffff,,,,,,0x0
group distances:
15 40 40 40
40 15 40 40
40 40 15 40
40 40 40 15
trying to group Group objects into misc objects according to physical
distances
found minimal distance 40 between objects
object 1 is minimally connected to 0
object 2 is minimally connected to 0
object 3 is minimally connected to 0
found transitive graph with 4 objects with minimal distance 40
=========================

The questions are:
1. What should I change in my patch to have it committed into the svn?
Specifically:
- where do I put IBM-specific code?
- may be there is a better way to detect that no cache info was fetched
from sysfs
- is the coding style ok? :-)

2. Do not I miss something in my patch in order to solve the problems
mentioned in the beginning of this mail?

Thank you.