Open MPI logo

Hardware Locality Development Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Hardware Locality Development mailing list

Subject: Re: [hwloc-devel] Strange results on itanium 2
From: Brice Goglin (Brice.Goglin_at_[hidden])
Date: 2010-10-29 18:24:14


Le 30/10/2010 00:01, Jirka Hladky a écrit :
> On Friday, October 29, 2010 10:59:25 pm Brice Goglin wrote:
>
>> Le 29/10/2010 21:57, Jirka Hladky a écrit :
>>
>>> Hi Samuel,
>>>
>>> I have attached the output of
>>> tests/linux/gather-topology.sh `uname --kernel-release`_`uname --
>>> nodename`_gather-topology
>>>
>>> I'm sorry for the long delay - systems has been used by somebody else, I
>>> had to wait for it to be free.
>>>
>>> System is running kernel 2.6.18-227.el5 (RHEL 5.6). ia64 is not supported
>>> on RHEL 6.0 so I cannot really test it on the new kernel.
>>>
>>> It would be really interesting if you can recognize if it's a kernel bug
>>> or hwloc problem.
>>>
>> /sys/devices/system/cpu/cpu*/cache/index4/shared_cpu_map contains an
>> empty map. This index4 is a L3 cache. But this map means that this cache
>> is near none of the cores... The instruction L2 has the same problem
>> (index3 instead of 4). This is a kernel bug.
>>
>> But, we already have a dedicated work-around in hwloc
>> (src/topology-linux.c):
>> if (hwloc_bitmap_weight(cacheset) < 1)
>> /* mask is wrong (happens on ia64), assumes it's not shared */
>> hwloc_bitmap_only(cacheset, i);
>>
>> This work-around worked fine on old itaniums since they had one L3, one
>> L2 and one L1 per core. Your machine has hyperthreading, so our
>> work-around creates one L3 per thread, while L1 and L2 (properly
>> reported by the kernel) are core-specific. Maybe hwloc should just
>> ignore caches with invalid shared_cpu_map.
>>
>> Brice
>>
>> _______________________________________________
>> hwloc-devel mailing list
>> hwloc-devel_at_[hidden]
>> http://www.open-mpi.org/mailman/listinfo.cgi/hwloc-devel
>>
>
> Hi Brice,
>
> thanks for looking into it! I'm going to open a BZ for it and put you on the
> Cc.
>

Thanks.

> BTW, is there some documentation on /sys/devices/system/cpu/* tree?
>

There's some doc in Documentation/ABI/*/sysfs-devices-* inside the
kernel source tree.

Brice