Open MPI logo

Hardware Locality Development Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Hardware Locality Development mailing list

Subject: [hwloc-devel] PUs not located under cores
From: Jeff Squyres (jsquyres_at_[hidden])
Date: 2010-06-24 12:02:48


I have another problem report for OMPI hwloc support from IBM -- they have a PPC-based machine with output that looks like this:

-----
[root_at_p6ihhpc5 tests]# hwloc-info
depth 0: 1 Machine (type #1)
 depth 1: 4 NUMANodes (type #2)
  depth 2: 64 PUs (type #6)
[root_at_p6ihhpc5 tests]# hwloc-ls
Machine (123GB)
  NUMANode #0 (phys=0 30GB)
    PU #0 (phys=0)
    PU #1 (phys=1)
    PU #2 (phys=2)
    PU #3 (phys=3)
    PU #4 (phys=4)
    PU #5 (phys=5)
    PU #6 (phys=6)
    PU #7 (phys=7)
    PU #8 (phys=8)
    PU #9 (phys=9)
    PU #10 (phys=10)
    PU #11 (phys=11)
    PU #12 (phys=12)
    PU #13 (phys=13)
    PU #14 (phys=14)
    PU #15 (phys=15)
  NUMANode #1 (phys=1 31GB)
    PU #16 (phys=16)
    PU #17 (phys=17)
    PU #18 (phys=18)
    PU #19 (phys=19)
    PU #20 (phys=20)
    PU #21 (phys=21)
    PU #22 (phys=22)
    PU #23 (phys=23)
    PU #24 (phys=24)
    PU #25 (phys=25)
    PU #26 (phys=26)
    PU #27 (phys=27)
    PU #28 (phys=28)
    PU #29 (phys=29)
    PU #30 (phys=30)
    PU #31 (phys=31)
  NUMANode #2 (phys=2 31GB)
    PU #32 (phys=32)
    PU #33 (phys=33)
    PU #34 (phys=34)
    PU #35 (phys=35)
    PU #36 (phys=36)
    PU #37 (phys=37)
    PU #38 (phys=38)
    PU #39 (phys=39)
    PU #40 (phys=40)
    PU #41 (phys=41)
    PU #42 (phys=42)
    PU #43 (phys=43)
    PU #44 (phys=44)
    PU #45 (phys=45)
    PU #46 (phys=46)
    PU #47 (phys=47)
  NUMANode #3 (phys=3 31GB)
    PU #48 (phys=48)
    PU #49 (phys=49)
    PU #50 (phys=50)
    PU #51 (phys=51)
    PU #52 (phys=52)
    PU #53 (phys=53)
    PU #54 (phys=54)
    PU #55 (phys=55)
    PU #56 (phys=56)
    PU #57 (phys=57)
    PU #58 (phys=58)
    PU #59 (phys=59)
    PU #60 (phys=60)
    PU #61 (phys=61)
    PU #62 (phys=62)
    PU #63 (phys=63)
-----

Is that right? I.e., is it proper that hwloc can return just a pile of PUs under a numa node with no core as an umbrella?

I can certainly code around this if we expect this kind of output from hwloc -- the current code using hwloc topology info in OMPI assumes that PUs will be under cores. But before I did that, I wanted to ask if this was expected.

It might be worth adding a section in hwloc.doxy describing stuff like this (yes, I'm volunteering to write it :-) ) -- that using hwloc can still result in lots of different kinds of outputs, depending on the type of machine you're running on. Case in point is the above output vs. the output from a typical intel/amd-based machine where you have a topology containing a machine, sockets, various caches, numa nodes, cores, and PU's.

-- 
Jeff Squyres
jsquyres_at_[hidden]
For corporate legal information go to:
http://www.cisco.com/web/about/doing_business/legal/cri/