Just as a follow up to this thread. I spoke with someone from IBM and they tell me that 2 cores of 4 hardware threads each are hidden from the kernel (how do they do that?) and used for the custom HS4 cards we have installed on the IO nodes, which explains why I see only 60 instead of 68 threads. the 2 bgvrnic tasks I see spinning at 100% run on threads 58/59 and service the connection from ION to CN.
It looks as though everything is reporting as expected - as long as I compile hwloc on the ION itself, it seems to be correct.
Thanks and sorry for any misunderstanding
> -----Original Message-----
> From: hwloc-users [mailto:hwloc-users-bounces_at_[hidden]] On Behalf
> Of Chris Samuel
> Sent: 26 March 2014 13:42
> To: Hardware locality user list
> Subject: Re: [hwloc-users] BGQ question.
> On Wed, 26 Mar 2014 11:56:08 AM Biddiscombe, John A. wrote:
> > I canât test this as the system is down for maintenance, but if memory
> > serves me correctly, the GCC compiled lstopo also showed 60 cores
> > instead of 64/68.
> It can only report what the kernel reports and it appears your kernel is not
> reporting the same number of cores on an IO node as ours.
> It would be interesting to compare kernel version and boot command line.
> Ours are:
> -bash-4.1# uname -a
> Linux r00-id-j01.pcf.vlsci.unimelb.edu.au 2.6.32-
> 279.14.1.bgq.el6_V1R2M0_36.ppc64 #1 SMP Tue Jun 11 15:50:53 CDT 2013
> ppc64 ppc64 ppc64 GNU/Linux
> -bash-4.1# cat /proc/cmdline
> root=/dev/ram0 rdinit=/init raid=noautodetect loglevel=5
> This is the end of our /proc/cpuinfo showing 68 hardware threads
> (17 cores exposed).
> -bash-4.1# tail -n 9 /proc/cpuinfo
> processor : 67
> cpu : A2 (Blue Gene/Q)
> clock : 1600.000000MHz
> revision : 2.0 (pvr 0049 0200)
> timebase : 1600000000
> platform : Blue Gene/Q
> model : ibm,bluegeneq
> > I am not certain if this gcc was in any was âspecialâ for bgq.
> There is a GCC cross compiler, but it's not the /usr/bin/gcc one.
> Christopher Samuel Senior Systems Administrator
> VLSCI - Victorian Life Sciences Computation Initiative
> Email: samuel_at_[hidden] Phone: +61 (0)3 903 55545
> http://www.vlsci.org.au/ http://twitter.com/vlsci
> hwloc-users mailing list