Open MPI logo

Hardware Locality Users' Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Hardware Locality Users mailing list

Subject: Re: [hwloc-users] BGQ question.
From: Biddiscombe, John A. (biddisco_at_[hidden])
Date: 2014-03-28 06:04:29


Just as a follow up to this thread. I spoke with someone from IBM and they tell me that 2 cores of 4 hardware threads each are hidden from the kernel (how do they do that?) and used for the custom HS4 cards we have installed on the IO nodes, which explains why I see only 60 instead of 68 threads. the 2 bgvrnic tasks I see spinning at 100% run on threads 58/59 and service the connection from ION to CN.

It looks as though everything is reporting as expected - as long as I compile hwloc on the ION itself, it seems to be correct.

Thanks and sorry for any misunderstanding

JB


> -----Original Message-----
> From: hwloc-users [mailto:hwloc-users-bounces_at_[hidden]] On Behalf
> Of Chris Samuel
> Sent: 26 March 2014 13:42
> To: Hardware locality user list
> Subject: Re: [hwloc-users] BGQ question.
>
> On Wed, 26 Mar 2014 11:56:08 AM Biddiscombe, John A. wrote:
>
> > I can’t test this as the system is down for maintenance, but if memory
> > serves me correctly, the GCC compiled lstopo also showed 60 cores
> > instead of 64/68.
>
> It can only report what the kernel reports and it appears your kernel is not
> reporting the same number of cores on an IO node as ours.
>
> It would be interesting to compare kernel version and boot command line.
>
> Ours are:
>
> -bash-4.1# uname -a
> Linux r00-id-j01.pcf.vlsci.unimelb.edu.au 2.6.32-
> 279.14.1.bgq.el6_V1R2M0_36.ppc64 #1 SMP Tue Jun 11 15:50:53 CDT 2013
> ppc64 ppc64 ppc64 GNU/Linux
>
>
> -bash-4.1# cat /proc/cmdline
> root=/dev/ram0 rdinit=/init raid=noautodetect loglevel=5
>
>
> This is the end of our /proc/cpuinfo showing 68 hardware threads
> (17 cores exposed).
>
> -bash-4.1# tail -n 9 /proc/cpuinfo
>
> processor : 67
> cpu : A2 (Blue Gene/Q)
> clock : 1600.000000MHz
> revision : 2.0 (pvr 0049 0200)
>
> timebase : 1600000000
> platform : Blue Gene/Q
> model : ibm,bluegeneq
>
>
> > I am not certain if this gcc was in any was ‘special’ for bgq.
>
> There is a GCC cross compiler, but it's not the /usr/bin/gcc one.
>
> cheers!
> Chris
> --
> Christopher Samuel Senior Systems Administrator
> VLSCI - Victorian Life Sciences Computation Initiative
> Email: samuel_at_[hidden] Phone: +61 (0)3 903 55545
> http://www.vlsci.org.au/ http://twitter.com/vlsci
>
> _______________________________________________
> hwloc-users mailing list
> hwloc-users_at_[hidden]
> http://www.open-mpi.org/mailman/listinfo.cgi/hwloc-users