Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: Re: [OMPI users] "-bind-to numa" of openmpi-1.7.4rc1 dosen't work for our magny cours based 32 core node
From: tmishima_at_[hidden]
Date: 2013-12-20 23:09:08


Hi Ralph,

Thank you very much. I tried many things such as:

mpirun -np 2 -host node05 -report-bindings -mca rmaps lama -mca
rmaps_lama_bind 1c myprog

But every try failed. At least they were accepted by openmpi-1.7.3 as far
as I remember.
Anyway, please check it when you have a time, because using lama comes from
my curiosity.

Regards,
Tetsuya Mishima

> I'll try to take a look at it - my expectation is that lama might get
stuck because you didn't tell it a pattern to map, and I doubt that code
path has seen much testing.
>
>
> On Dec 20, 2013, at 5:52 PM, tmishima_at_[hidden] wrote:
>
> >
> >
> > Hi Ralph, I'm glad to hear that, thanks.
> >
> > By the way, yesterday I tried to check how lama in 1.7.4rc treat numa
node.
> >
> > Then, even wiht this simple command line, it freezed without any
massage:
> >
> > mpirun -np 2 -host node05 -mca rmaps lama myprog
> >
> > Could you check what happened?
> >
> > Is it better to open new thread or continue this thread?
> >
> > Regards,
> > Tetsuya Mishima
> >
> >
> >> I'll make it work so that NUMA can be either above or below socket
> >>
> >> On Dec 20, 2013, at 2:57 AM, tmishima_at_[hidden] wrote:
> >>
> >>>
> >>>
> >>> Hi Brice,
> >>>
> >>> Thank you for your comment. I understand what you mean.
> >>>
> >>> My opinion was made just considering easy way to adjust the code for
> >>> inversion of hierarchy in object tree.
> >>>
> >>> Tetsuya Mishima
> >>>
> >>>
> >>>> I don't think there's any such difference.
> >>>> Also, all these NUMA architectures are reported the same by hwloc,
and
> >>>> therefore used the same in Open MPI.
> >>>>
> >>>> And yes, L3 and NUMA are topologically-identical on AMD Magny-Cours
> > (and
> >>>> most recent AMD and Intel platforms).
> >>>>
> >>>> Brice
> >>>>
> >>>>
> >>>>
> >>>> Le 20/12/2013 11:33, tmishima_at_[hidden] a écrit :
> >>>>>
> >>>>> Hi Ralph,
> >>>>>
> >>>>> The numa-node in AMD Mangy-Cours/Interlagos is so called cc(cache
> >>>>> coherent)NUMA,
> >>>>> which seems to be a little bit different from the traditional numa
> >>> defined
> >>>>> in openmpi.
> >>>>>
> >>>>> I notice that ccNUMA object is almost same as L3cache object.
> >>>>> So "-bind-to l3cache" or "-map-by l3cache" is valid for what I want
> > to
> >>> do.
> >>>>> Therefore, "do not touch it" is one of the solution, I think ...
> >>>>>
> >>>>> Anyway, mixing up these two types of numa is the problem.
> >>>>>
> >>>>> Regards,
> >>>>> Tetsuya Mishima
> >>>>>
> >>>>>> I can wait it'll be fixed in 1.7.5 or later, because putting
> > "-bind-to
> >>>>>> numa"
> >>>>>> and "-map-by numa" at the same time works as a workaround.
> >>>>>>
> >>>>>> Thanks,
> >>>>>> Tetsuya Mishima
> >>>>>>
> >>>>>>> Yeah, it will impact everything that uses hwloc topology maps, I
> >>> fear.
> >>>>>>>
> >>>>>>> One side note: you'll need to add --hetero-nodes to your cmd
line.
> > If
> >>>>> we
> >>>>>> don't see that, we assume that all the node topologies are
identical
> > -
> >>>>>> which clearly isn't true here.
> >>>>>>> I'll try to resolve the hier inversion over the holiday - won't
be
> >>> for
> >>>>>> 1.7.4, but hopefully for 1.7.5
> >>>>>>> Thanks
> >>>>>>> Ralph
> >>>>>>>
> >>>>>>> On Dec 18, 2013, at 9:44 PM, tmishima_at_[hidden] wrote:
> >>>>>>>
> >>>>>>>>
> >>>>>>>> I think it's normal for AMD opteron having 8/16 cores such as
> >>>>>>>> magny cours or interlagos. Because it usually has 2 numa nodes
> >>>>>>>> in a cpu(socket), numa-node can not include a socket. This type
> >>>>>>>> of hierarchy would be natural.
> >>>>>>>>
> >>>>>>>> (node03 is Dell PowerEdge R815 and maybe quite common, I guess)
> >>>>>>>>
> >>>>>>>> By the way, I think this inversion should affect rmaps_lama
> > mapping.
> >>>>>>>>
> >>>>>>>> Tetsuya Mishima
> >>>>>>>>
> >>>>>>>>> Ick - yeah, that would be a problem. I haven't seen that type
of
> >>>>>>>> hierarchical inversion before - is node03 a different type of
> > chip?
> >>>>>>>>> Might take awhile for me to adjust the code to handle hier
> >>>>>>>> inversion... :-(
> >>>>>>>>> On Dec 18, 2013, at 9:05 PM, tmishima_at_[hidden] wrote:
> >>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> Hi Ralph,
> >>>>>>>>>>
> >>>>>>>>>> I found the reason. I attached the main part of output with 32
> >>>>>>>>>> core node(node03) and 8 core node(node05) at the bottom.
> >>>>>>>>>>
> >>>>>>>>>> From this information, socket of node03 includes numa-node.
> >>>>>>>>>> On the other hand, numa-node of node05 includes socket.
> >>>>>>>>>> The direction of object tree is opposite.
> >>>>>>>>>>
> >>>>>>>>>> Since "-map-by socket" may be assumed as default,
> >>>>>>>>>> for node05, "-bind-to numa and -map-by socket" means
> >>>>>>>>>> upward search. For node03, this should be downward.
> >>>>>>>>>>
> >>>>>>>>>> I guess that openmpi-1.7.4rc1 will always assume numa-node
> >>>>>>>>>> includes socket. Is it right? Then, upward search is assumed
> >>>>>>>>>> in orte_rmaps_base_compute_bindings even for node03 when I
> >>>>>>>>>> put "-bind-to numa and -map-by socket" option.
> >>>>>>>>>>
> >>>>>>>>>> [node03.cluster:15508] [[38286,0],0] rmaps:base:compute_usage
> >>>>>>>>>> [node03.cluster:15508] mca:rmaps: compute bindings for job
> >>>>> [38286,1]
> >>>>>>>> with
> >>>>>>>>>> policy NUMA
> >>>>>>>>>> [node03.cluster:15508] mca:rmaps: bind upwards for job
[38286,1]
> >>>>> with
> >>>>>>>>>> bindings NUMA
> >>>>>>>>>> [node03.cluster:15508] [[38286,0],0] bind:upward target
NUMANode
> >>>>> type
> >>>>>>>>>> Machine
> >>>>>>>>>>
> >>>>>>>>>> That's the reason of this trouble. Therefore, adding "-map-by
> >>> core"
> >>>>>>>> works.
> >>>>>>>>>> (mapping pattern seems to be strange ...)
> >>>>>>>>>>
> >>>>>>>>>> [mishima_at_node03 demos]$ mpirun -np 8 -bind-to numa -map-by
core
> >>>>>>>>>> -report-bindings myprog
> >>>>>>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target
NUMANode
> >>>>> type
> >>>>>>>> Cache
> >>>>>>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target
NUMANode
> >>>>> type
> >>>>>>>> Cache
> >>>>>>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target
NUMANode
> >>>>> type
> >>>>>>>> Cache
> >>>>>>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target
NUMANode
> >>>>> type
> >>>>>>>>>> NUMANode
> >>>>>>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target
NUMANode
> >>>>> type
> >>>>>>>> Cache
> >>>>>>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target
NUMANode
> >>>>> type
> >>>>>>>> Cache
> >>>>>>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target
NUMANode
> >>>>> type
> >>>>>>>> Cache
> >>>>>>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target
NUMANode
> >>>>> type
> >>>>>>>>>> NUMANode
> >>>>>>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target
NUMANode
> >>>>> type
> >>>>>>>> Cache
> >>>>>>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target
NUMANode
> >>>>> type
> >>>>>>>> Cache
> >>>>>>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target
NUMANode
> >>>>> type
> >>>>>>>> Cache
> >>>>>>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target
NUMANode
> >>>>> type
> >>>>>>>>>> NUMANode
> >>>>>>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target
NUMANode
> >>>>> type
> >>>>>>>> Cache
> >>>>>>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target
NUMANode
> >>>>> type
> >>>>>>>> Cache
> >>>>>>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target
NUMANode
> >>>>> type
> >>>>>>>> Cache
> >>>>>>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target
NUMANode
> >>>>> type
> >>>>>>>>>> NUMANode
> >>>>>>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target
NUMANode
> >>>>> type
> >>>>>>>> Cache
> >>>>>>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target
NUMANode
> >>>>> type
> >>>>>>>> Cache
> >>>>>>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target
NUMANode
> >>>>> type
> >>>>>>>> Cache
> >>>>>>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target
NUMANode
> >>>>> type
> >>>>>>>>>> NUMANode
> >>>>>>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target
NUMANode
> >>>>> type
> >>>>>>>> Cache
> >>>>>>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target
NUMANode
> >>>>> type
> >>>>>>>> Cache
> >>>>>>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target
NUMANode
> >>>>> type
> >>>>>>>> Cache
> >>>>>>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target
NUMANode
> >>>>> type
> >>>>>>>>>> NUMANode
> >>>>>>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target
NUMANode
> >>>>> type
> >>>>>>>> Cache
> >>>>>>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target
NUMANode
> >>>>> type
> >>>>>>>> Cache
> >>>>>>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target
NUMANode
> >>>>> type
> >>>>>>>> Cache
> >>>>>>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target
NUMANode
> >>>>> type
> >>>>>>>>>> NUMANode
> >>>>>>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target
NUMANode
> >>>>> type
> >>>>>>>> Cache
> >>>>>>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target
NUMANode
> >>>>> type
> >>>>>>>> Cache
> >>>>>>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target
NUMANode
> >>>>> type
> >>>>>>>> Cache
> >>>>>>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target
NUMANode
> >>>>> type
> >>>>>>>>>> NUMANode
> >>>>>>>>>> [node03.cluster:15885] MCW rank 2 bound to socket 0[core 0[hwt
> >>> 0]],
> >>>>>>>> socket
> >>>>>>>>>> 0[core 1[hwt 0]], socket 0[core 2[hwt 0]], so
> >>>>>>>>>> cket 0[core 3[hwt 0]]:
> >>>>>>>>>>
> >>>>>
[B/B/B/B/./././.][./././././././.][./././././././.][./././././././.]
> >>>>>>>>>> [node03.cluster:15885] MCW rank 3 bound to socket 0[core 0[hwt
> >>> 0]],
> >>>>>>>> socket
> >>>>>>>>>> 0[core 1[hwt 0]], socket 0[core 2[hwt 0]], so
> >>>>>>>>>> cket 0[core 3[hwt 0]]:
> >>>>>>>>>>
> >>>>>
[B/B/B/B/./././.][./././././././.][./././././././.][./././././././.]
> >>>>>>>>>> [node03.cluster:15885] MCW rank 4 bound to socket 0[core 4[hwt
> >>> 0]],
> >>>>>>>> socket
> >>>>>>>>>> 0[core 5[hwt 0]], socket 0[core 6[hwt 0]], so
> >>>>>>>>>> cket 0[core 7[hwt 0]]:
> >>>>>>>>>>
> >>>>>
[././././B/B/B/B][./././././././.][./././././././.][./././././././.]
> >>>>>>>>>> [node03.cluster:15885] MCW rank 5 bound to socket 0[core 4[hwt
> >>> 0]],
> >>>>>>>> socket
> >>>>>>>>>> 0[core 5[hwt 0]], socket 0[core 6[hwt 0]], so
> >>>>>>>>>> cket 0[core 7[hwt 0]]:
> >>>>>>>>>>
> >>>>>
[././././B/B/B/B][./././././././.][./././././././.][./././././././.]
> >>>>>>>>>> [node03.cluster:15885] MCW rank 6 bound to socket 0[core 4[hwt
> >>> 0]],
> >>>>>>>> socket
> >>>>>>>>>> 0[core 5[hwt 0]], socket 0[core 6[hwt 0]], so
> >>>>>>>>>> cket 0[core 7[hwt 0]]:
> >>>>>>>>>>
> >>>>>
[././././B/B/B/B][./././././././.][./././././././.][./././././././.]
> >>>>>>>>>> [node03.cluster:15885] MCW rank 7 bound to socket 0[core 4[hwt
> >>> 0]],
> >>>>>>>> socket
> >>>>>>>>>> 0[core 5[hwt 0]], socket 0[core 6[hwt 0]], so
> >>>>>>>>>> cket 0[core 7[hwt 0]]:
> >>>>>>>>>>
> >>>>>
[././././B/B/B/B][./././././././.][./././././././.][./././././././.]
> >>>>>>>>>> [node03.cluster:15885] MCW rank 0 bound to socket 0[core 0[hwt
> >>> 0]],
> >>>>>>>> socket
> >>>>>>>>>> 0[core 1[hwt 0]], socket 0[core 2[hwt 0]], so
> >>>>>>>>>> cket 0[core 3[hwt 0]]:
> >>>>>>>>>>
> >>>>>
[B/B/B/B/./././.][./././././././.][./././././././.][./././././././.]
> >>>>>>>>>> [node03.cluster:15885] MCW rank 1 bound to socket 0[core 0[hwt
> >>> 0]],
> >>>>>>>> socket
> >>>>>>>>>> 0[core 1[hwt 0]], socket 0[core 2[hwt 0]], so
> >>>>>>>>>> cket 0[core 3[hwt 0]]:
> >>>>>>>>>>
> >>>>>
[B/B/B/B/./././.][./././././././.][./././././././.][./././././././.]
> >>>>>>>>>> Hello world from process 6 of 8
> >>>>>>>>>> Hello world from process 5 of 8
> >>>>>>>>>> Hello world from process 0 of 8
> >>>>>>>>>> Hello world from process 7 of 8
> >>>>>>>>>> Hello world from process 3 of 8
> >>>>>>>>>> Hello world from process 4 of 8
> >>>>>>>>>> Hello world from process 2 of 8
> >>>>>>>>>> Hello world from process 1 of 8
> >>>>>>>>>>
> >>>>>>>>>> Regards,
> >>>>>>>>>> Tetsuya Mishima
> >>>>>>>>>>
> >>>>>>>>>> [node03.cluster:15508] Type: Machine Number of child objects:
4
> >>>>>>>>>> Name=NULL
> >>>>>>>>>> total=132358820KB
> >>>>>>>>>> Backend=Linux
> >>>>>>>>>> OSName=Linux
> >>>>>>>>>> OSRelease=2.6.18-308.16.1.el5
> >>>>>>>>>> OSVersion="#1 SMP Tue Oct 2 22:01:43 EDT 2012"
> >>>>>>>>>> Architecture=x86_64
> >>>>>>>>>> Cpuset: 0xffffffff
> >>>>>>>>>> Online: 0xffffffff
> >>>>>>>>>> Allowed: 0xffffffff
> >>>>>>>>>> Bind CPU proc: TRUE
> >>>>>>>>>> Bind CPU thread: TRUE
> >>>>>>>>>> Bind MEM proc: FALSE
> >>>>>>>>>> Bind MEM thread: TRUE
> >>>>>>>>>> Type: Socket Number of child objects: 2
> >>>>>>>>>> Name=NULL
> >>>>>>>>>> total=33071780KB
> >>>>>>>>>> CPUModel="AMD Opteron(tm) Processor 6136"
> >>>>>>>>>> Cpuset: 0x000000ff
> >>>>>>>>>> Online: 0x000000ff
> >>>>>>>>>> Allowed: 0x000000ff
> >>>>>>>>>> Type: NUMANode Number of child objects: 1
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> [node05.cluster:21750] Type: Machine Number of child objects:
2
> >>>>>>>>>> Name=NULL
> >>>>>>>>>> total=33080072KB
> >>>>>>>>>> Backend=Linux
> >>>>>>>>>> OSName=Linux
> >>>>>>>>>> OSRelease=2.6.18-308.16.1.el5
> >>>>>>>>>> OSVersion="#1 SMP Tue Oct 2 22:01:43 EDT 2012"
> >>>>>>>>>> Architecture=x86_64
> >>>>>>>>>> Cpuset: 0x000000ff
> >>>>>>>>>> Online: 0x000000ff
> >>>>>>>>>> Allowed: 0x000000ff
> >>>>>>>>>> Bind CPU proc: TRUE
> >>>>>>>>>> Bind CPU thread: TRUE
> >>>>>>>>>> Bind MEM proc: FALSE
> >>>>>>>>>> Bind MEM thread: TRUE
> >>>>>>>>>> Type: NUMANode Number of child objects: 1
> >>>>>>>>>> Name=NULL
> >>>>>>>>>> local=16532232KB
> >>>>>>>>>> total=16532232KB
> >>>>>>>>>> Cpuset: 0x0000000f
> >>>>>>>>>> Online: 0x0000000f
> >>>>>>>>>> Allowed: 0x0000000f
> >>>>>>>>>> Type: Socket Number of child objects: 1
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>> Hmm...try adding "-mca rmaps_base_verbose 10 -mca
> >>> ess_base_verbose
> >>>>>> 5"
> >>>>>>>> to
> >>>>>>>>>> your cmd line and let's see what it thinks it found.
> >>>>>>>>>>>
> >>>>>>>>>>> On Dec 18, 2013, at 6:55 PM, tmishima_at_[hidden]
wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> Hi, I report one more problem with openmpi-1.7.4rc1,
> >>>>>>>>>>>> which is more serious.
> >>>>>>>>>>>>
> >>>>>>>>>>>> For our 32 core nodes(AMD magny cours based) which has
> >>>>>>>>>>>> 8 numa-nodes, "-bind-to numa" does not work. Without
> >>>>>>>>>>>> this option, it works. For your infomation, at the
> >>>>>>>>>>>> bottom of this mail, I added the lstopo information
> >>>>>>>>>>>> of the node.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Regards,
> >>>>>>>>>>>> Tetsuya Mishima
> >>>>>>>>>>>>
> >>>>>>>>>>>> [mishima_at_manage ~]$ qsub -I -l nodes=1:ppn=32
> >>>>>>>>>>>> qsub: waiting for job 8352.manage.cluster to start
> >>>>>>>>>>>> qsub: job 8352.manage.cluster ready
> >>>>>>>>>>>>
> >>>>>>>>>>>> [mishima_at_node03 demos]$ mpirun -np 8 -report-bindings
-bind-to
> >>>>> numa
> >>>>>>>>>> myprog
> >>>>>>>>>>>> [node03.cluster:15316] [[37582,0],0] bind:upward target
> > NUMANode
> >>>>>> type
> >>>>>>>>>>>> Machine
> >>>>>>>>>>>>
> >>>>>
> >>>
> >
--------------------------------------------------------------------------
> >>>>>>>>>>>> A request was made to bind to NUMA, but an appropriate
target
> >>>>> could
> >>>>>>>> not
> >>>>>>>>>>>> be found on node node03.
> >>>>>>>>>>>>
> >>>>>
> >>>
> >
--------------------------------------------------------------------------
> >>>>>>>>>>>> [mishima_at_node03 ~]$ cd ~/Desktop/openmpi-1.7/demos/
> >>>>>>>>>>>> [mishima_at_node03 demos]$ mpirun -np 8 -report-bindings myprog
> >>>>>>>>>>>> [node03.cluster:15282] MCW rank 2 bound to socket 1[core 8
[hwt
> >>>>> 0]]:
> >>>>>>>>>>>> [./././././././.][B/././././././.][./././././././.][
> >>>>>>>>>>>> ./././././././.]>>>>>>>>>>>> [node03.cluster:15282] MCW rank
3 bound to socket 1[core 9[hwt
> >>>>> 0]]:
> >>>>>>>>>>>> [./././././././.][./B/./././././.][./././././././.][
> >>>>>>>>>>>> ./././././././.]
> >>>>>>>>>>>> [node03.cluster:15282] MCW rank 4 bound to socket 2[core 16
> > [hwt
> >>>>>> 0]]:
> >>>>>>>>>>>> [./././././././.][./././././././.][B/././././././.]
> >>>>>>>>>>>> [./././././././.]
> >>>>>>>>>>>> [node03.cluster:15282] MCW rank 5 bound to socket 2[core 17
> > [hwt
> >>>>>> 0]]:
> >>>>>>>>>>>> [./././././././.][./././././././.][./B/./././././.]
> >>>>>>>>>>>> [./././././././.]
> >>>>>>>>>>>> [node03.cluster:15282] MCW rank 6 bound to socket 3[core 24
> > [hwt
> >>>>>> 0]]:
> >>>>>>>>>>>> [./././././././.][./././././././.][./././././././.]
> >>>>>>>>>>>> [B/././././././.]
> >>>>>>>>>>>> [node03.cluster:15282] MCW rank 7 bound to socket 3[core 25
> > [hwt
> >>>>>> 0]]:
> >>>>>>>>>>>> [./././././././.][./././././././.][./././././././.]
> >>>>>>>>>>>> [./B/./././././.]
> >>>>>>>>>>>> [node03.cluster:15282] MCW rank 0 bound to socket 0[core 0
[hwt
> >>>>> 0]]:
> >>>>>>>>>>>> [B/././././././.][./././././././.][./././././././.][
> >>>>>>>>>>>> ./././././././.]
> >>>>>>>>>>>> [node03.cluster:15282] MCW rank 1 bound to socket 0[core 1
[hwt
> >>>>> 0]]:
> >>>>>>>>>>>> [./B/./././././.][./././././././.][./././././././.][
> >>>>>>>>>>>> ./././././././.]
> >>>>>>>>>>>> Hello world from process 2 of 8
> >>>>>>>>>>>> Hello world from process 5 of 8
> >>>>>>>>>>>> Hello world from process 4 of 8
> >>>>>>>>>>>> Hello world from process 3 of 8>>>>>>>>>> Hello world from
> > process 1 of 8
> >>>>>>>>>>>> Hello world from process 7 of 8
> >>>>>>>>>>>> Hello world from process 6 of 8
> >>>>>>>>>>>> Hello world from process 0 of 8
> >>>>>>>>>>>> [mishima_at_node03 demos]$ ~/opt/hwloc/bin/lstopo-no-graphics
> >>>>>>>>>>>> Machine (126GB)
> >>>>>>>>>>>> Socket L#0 (32GB)
> >>>>>>>>>>>> NUMANode L#0 (P#0 16GB) + L3 L#0 (5118KB)
> >>>>>>>>>>>> L2 L#0 (512KB) + L1d L#0 (64KB) + L1i L#0 (64KB) + Core L#0
> > +
> >>>>> PU
> >>>>>>>>>> L#0
> >>>>>>>>>>>> (P#0)
> >>>>>>>>>>>> L2 L#1 (512KB) + L1d L#1 (64KB) + L1i L#1 (64KB) + Core L#1
> > +
> >>>>> PU
> >>>>>>>>>> L#1
> >>>>>>>>>>>> (P#1)
> >>>>>>>>>>>> L2 L#2 (512KB) + L1d L#2 (64KB) + L1i L#2 (64KB) + Core L#2
> > +
> >>>>> PU
> >>>>>>>>>> L#2
> >>>>>>>>>>>> (P#2)
> >>>>>>>>>>>> L2 L#3 (512KB) + L1d L#3 (64KB) + L1i L#3 (64KB) + Core L#3
> > +
> >>>>> PU
> >>>>>>>>>> L#3
> >>>>>>>>>>>> (P#3)
> >>>>>>>>>>>> NUMANode L#1 (P#1 16GB) + L3 L#1 (5118KB)
> >>>>>>>>>>>> L2 L#4 (512KB) + L1d L#4 (64KB) + L1i L#4 (64KB) + Core L#4
> > +
> >>>>> PU
> >>>>>>>>>> L#4
> >>>>>>>>>>>> (P#4)
> >>>>>>>>>>>> L2 L#5 (512KB) + L1d L#5 (64KB) + L1i L#5 (64KB) + Core L#5
> > +
> >>>>> PU
> >>>>>>>>>> L#5
> >>>>>>>>>>>> (P#5)
> >>>>>>>>>>>> L2 L#6 (512KB) + L1d L#6 (64KB) + L1i L#6 (64KB) + Core L#6
> > +
> >>>>> PU
> >>>>>>>>>> L#6
> >>>>>>>>>>>> (P#6)
> >>>>>>>>>>>> L2 L#7 (512KB) + L1d L#7 (64KB) + L1i L#7 (64KB) + Core L#7
> > +
> >>>>> PU>>>>>> L#7
> >>>>>>>>>>>> (P#7)
> >>>>>>>>>>>> Socket L#1 (32GB)
> >>>>>>>>>>>> NUMANode L#2 (P#6 16GB) + L3 L#2 (5118KB)
> >>>>>>>>>>>> L2 L#8 (512KB) + L1d L#8 (64KB) + L1i L#8 (64KB) + Core L#8
> > +
> >>>>> PU
> >>>>>>>>>> L#8
> >>>>>>>>>>>> (P#8)
> >>>>>>>>>>>> L2 L#9 (512KB) + L1d L#9 (64KB) + L1i L#9 (64KB) + Core L#9
> > +
> >>>>> PU
> >>>>>>>>>> L#9
> >>>>>>>>>>>> (P#9)
> >>>>>>>>>>>> L2 L#10 (512KB) + L1d L#10 (64KB) + L1i L#10 (64KB) + Core
> >>>>> L#10
> >>>>>> +
> >>>>>>>>>> PU
> >>>>>>>>>>>> L#10 (P#10)
> >>>>>>>>>>>> L2 L#11 (512KB) + L1d L#11 (64KB) + L1i L#11 (64KB) + Core
> >>>>> L#11
> >>>>>> +
> >>>>>>>>>> PU
> >>>>>>>>>>>> L#11 (P#11)
> >>>>>>>>>>>> NUMANode L#3 (P#7 16GB) + L3 L#3 (5118KB)
> >>>>>>>>>>>> L2 L#12 (512KB) + L1d L#12 (64KB) + L1i L#12 (64KB) + Core
> >>>>> L#12
> >>>>>> +
> >>>>>>>>>> PU
> >>>>>>>>>>>> L#12 (P#12)
> >>>>>>>>>>>> L2 L#13 (512KB) + L1d L#13 (64KB) + L1i L#13 (64KB) + Core
> >>>>> L#13
> >>>>>> +
> >>>>>>>>>> PU
> >>>>>>>>>>>> L#13 (P#13)
> >>>>>>>>>>>> L2 L#14 (512KB) + L1d L#14 (64KB) + L1i L#14 (64KB) + Core
> >>>>> L#14
> >>>>>> +
> >>>>>>>>>> PU
> >>>>>>>>>>>> L#14 (P#14)
> >>>>>>>>>>>> L2 L#15 (512KB) + L1d L#15 (64KB) + L1i L#15 (64KB) + Core
> >>>>> L#15
> >>>>>> +
> >>>>>>>>>> PU
> >>>>>>>>>>>> L#15 (P#15)
> >>>>>>>>>>>> Socket L#2 (32GB)
> >>>>>>>>>>>> NUMANode L#4 (P#4 16GB) + L3 L#4 (5118KB)
> >>>>>>>>>>>> L2 L#16 (512KB) + L1d L#16 (64KB) + L1i L#16 (64KB) + Core
> >>>>> L#16
> >>>>>> +
> >>>>>>>>>> PU
> >>>>>>>>>>>> L#16 (P#16)
> >>>>>>>>>>>> L2 L#17 (512KB) + L1d L#17 (64KB) + L1i L#17 (64KB) + Core
> >>>>> L#17
> >>>>>> +
> >>>>>>>>>> PU
> >>>>>>>>>>>> L#17 (P#17)> >>>>> L2 L#18 (512KB) + L1d L#18 (64KB) +
L1i
> >>>>> L#18 (64KB) + Core L#18
> >>>>>> +
> >>>>>>>>>> PU
> >>>>>>>>>>>> L#18 (P#18)
> >>>>>>>>>>>> L2 L#19 (512KB) + L1d L#19 (64KB) + L1i L#19 (64KB) + Core
> >>>>> L#19
> >>>>>> +
> >>>>>>>>>> PU
> >>>>>>>>>>>> L#19 (P#19)
> >>>>>>>>>>>> NUMANode L#5 (P#5 16GB) + L3 L#5 (5118KB)
> >>>>>>>>>>>> L2 L#20 (512KB) + L1d L#20 (64KB) + L1i L#20 (64KB) + Core
> >>>>> L#20
> >>>>>> +
> >>>>>>>>>> PU
> >>>>>>>>>>>> L#20 (P#20)
> >>>>>>>>>>>> L2 L#21 (512KB) + L1d L#21 (64KB) + L1i L#21 (64KB) + Core
> >>>>> L#21
> >>>>>> +
> >>>>>>>>>> PU
> >>>>>>>>>>>> L#21 (P#21)
> >>>>>>>>>>>> L2 L#22 (512KB) + L1d L#22 (64KB) + L1i L#22 (64KB) + Core
> >>>>> L#22
> >>>>>> +
> >>>>>>>>>> PU
> >>>>>>>>>>>> L#22 (P#22)
> >>>>>>>>>>>> L2 L#23 (512KB) + L1d L#23 (64KB) + L1i L#23 (64KB) + Core
> >>>>> L#23
> >>>>>> +
> >>>>>>>>>> PU
> >>>>>>>>>>>> L#23 (P#23)
> >>>>>>>>>>>> Socket L#3 (32GB)
> >>>>>>>>>>>> NUMANode L#6 (P#2 16GB) + L3 L#6 (5118KB)
> >>>>>>>>>>>> L2 L#24 (512KB) + L1d L#24 (64KB) + L1i L#24 (64KB) + Core
> >>>>> L#24
> >>>>>> +
> >>>>>>>>>> PU
> >>>>>>>>>>>> L#24 (P#24)>>>>> L2 L#25 (512KB) + L1d L#25 (64KB) + L1i
> > L#25
> >>>>>> (64KB) + Core L#25 +
> >>>>>>>>>> PU
> >>>>>>>>>>>> L#25 (P#25)
> >>>>>>>>>>>> L2 L#26 (512KB) + L1d L#26 (64KB) + L1i L#26 (64KB) + Core
> >>>>> L#26
> >>>>>> +
> >>>>>>>>>> PU
> >>>>>>>>>>>> L#26 (P#26)
> >>>>>>>>>>>> L2 L#27 (512KB) + L1d L#27 (64KB) + L1i L#27 (64KB) + Core
> >>>>> L#27
> >>>>>> +
> >>>>>>>>>> PU
> >>>>>>>>>>>> L#27 (P#27)
> >>>>>>>>>>>> NUMANode L#7 (P#3 16GB) + L3 L#7 (5118KB)
> >>>>>>>>>>>> L2 L#28 (512KB) + L1d L#28 (64KB) + L1i L#28 (64KB) + Core
> >>>>> L#28
> >>>>>> +
> >>>>>>>>>> PU
> >>>>>>>>>>>> L#28 (P#28)
> >>>>>>>>>>>> L2 L#29 (512KB) + L1d L#29 (64KB) + L1i L#29 (64KB) + Core
> >>>>> L#29
> >>>>>> +
> >>>>>>>>>> PU
> >>>>>>>>>>>> L#29 (P#29)
> >>>>>>>>>>>> L2 L#30 (512KB) + L1d L#30 (64KB) + L1i L#30 (64KB) + Core
> >>>>> L#30
> >>>>>> +
> >>>>>>>>>> PU
> >>>>>>>>>>>> L#30 (P#30)
> >>>>>>>>>>>> L2 L#31 (512KB) + L1d L#31 (64KB) + L1i L#31 (64KB) + Core
> >>>>> L#31
> >>>>>> +
> >>>>>>>>>> PU
> >>>>>>>>>>>> L#31 (P#31)
> >>>>>>>>>>>> HostBridge L#0
> >>>>>>>>>>>> PCIBridge
> >>>>>>>>>>>> PCI 14e4:1639
> >>>>>>>>>>>> Net L#0 "eth0"
> >>>>>>>>>>>> PCI 14e4:1639
> >>>>>>>>>>>> Net L#1 "eth1"
> >>>>>>>>>>>> PCIBridge
> >>>>>>>>>>>> PCI 14e4:1639
> >>>>>>>>>>>> Net L#2 "eth2"
> >>>>>>>>>>>> PCI 14e4:1639
> >>>>>>>>>>>> Net L#3 "eth3"
> >>>>>>>>>>>> PCIBridge
> >>>>>>>>>>>> PCIBridge
> >>>>>>>>>>>> PCIBridge
> >>>>>>>>>>>> PCI 1000:0072
> >>>>>>>>>>>> Block L#4 "sdb"
> >>>>>>>>>>>> Block L#5 "sda"
> >>>>>>>>>>>> PCI 1002:4390
> >>>>>>>>>>>> Block L#6 "sr0"
> >>>>>>>>>>>> PCIBridge
> >>>>>>>>>>>> PCI 102b:0532
> >>>>>>>>>>>> HostBridge L#7
> >>>>>>>>>>>> PCIBridge
> >>>>>>>>>>>> PCI 15b3:6274
> >>>>>>>>>>>> Net L#7 "ib0"
> >>>>>>>>>>>> OpenFabrics L#8 "mthca0"
> >>>>>>>>>>>>
> >>>>>>>>>>>> _______________________________________________
> >>>>>>>>>>>> users mailing list
> >>>>>>>>>>>> users_at_[hidden]
> >>>>>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
> >>>>>>>>>>> _______________________________________________
> >>>>>>>>>>> users mailing list
> >>>>>>>>>>> users_at_[hidden]>>
> >>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
> >>>>>>>>>> _______________________________________________
> >>>>>>>>>> users mailing list
> >>>>>>>>>> users_at_[hidden]
> >>>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
> >>>>>>>>> _______________________________________________
> >>>>>>>>> users mailing list
> >>>>>>>>> users_at_[hidden]
> >>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
> >>>>>>>> _______________________________________________
> >>>>>>>> users mailing list
> >>>>>>>> users_at_[hidden]
> >>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
> >>>>>>> _______________________________________________
> >>>>>>> users mailing list
> >>>>>>> users_at_[hidden]
> >>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
> >>>>>> _______________________________________________
> >>>>>> users mailing list
> >>>>>> users_at_[hidden]
> >>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
> >>>>> _______________________________________________
> >>>>> users mailing list
> >>>>> users_at_[hidden]
> >>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
> >>>>
> >>>> _______________________________________________
> >>>> users mailing list
> >>>> users_at_[hidden]
> >>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
> >>>
> >>> _______________________________________________
> >>> users mailing list
> >>> users_at_[hidden]
> >>> http://www.open-mpi.org/mailman/listinfo.cgi/users
> >>
> >> _______________________________________________
> >> users mailing list
> >> users_at_[hidden]
> >> http://www.open-mpi.org/mailman/listinfo.cgi/users
> >
> > _______________________________________________
> > users mailing list
> > users_at_[hidden]
> > http://www.open-mpi.org/mailman/listinfo.cgi/users
>
> _______________________________________________
> users mailing list
> users_at_[hidden]
> http://www.open-mpi.org/mailman/listinfo.cgi/users