Open MPI logo

Hardware Locality Development Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Hardware Locality Development mailing list

Subject: Re: [hwloc-devel] Strange results on itanium 2
From: Brice Goglin (Brice.Goglin_at_[hidden])
Date: 2010-10-29 16:59:25


Le 29/10/2010 21:57, Jirka Hladky a écrit :
> Hi Samuel,
>
> I have attached the output of
> tests/linux/gather-topology.sh `uname --kernel-release`_`uname --
> nodename`_gather-topology
>
> I'm sorry for the long delay - systems has been used by somebody else, I had
> to wait for it to be free.
>
> System is running kernel 2.6.18-227.el5 (RHEL 5.6). ia64 is not supported on
> RHEL 6.0 so I cannot really test it on the new kernel.
>
> It would be really interesting if you can recognize if it's a kernel bug or
> hwloc problem.
>

/sys/devices/system/cpu/cpu*/cache/index4/shared_cpu_map contains an
empty map. This index4 is a L3 cache. But this map means that this cache
is near none of the cores... The instruction L2 has the same problem
(index3 instead of 4). This is a kernel bug.

But, we already have a dedicated work-around in hwloc
(src/topology-linux.c):
          if (hwloc_bitmap_weight(cacheset) < 1)
            /* mask is wrong (happens on ia64), assumes it's not shared */
            hwloc_bitmap_only(cacheset, i);

This work-around worked fine on old itaniums since they had one L3, one
L2 and one L1 per core. Your machine has hyperthreading, so our
work-around creates one L3 per thread, while L1 and L2 (properly
reported by the kernel) are core-specific. Maybe hwloc should just
ignore caches with invalid shared_cpu_map.

Brice