Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: Re: [OMPI users] "-bind-to numa" of openmpi-1.7.4rc1 dosen't work for our magny cours based 32 core node
From: Ralph Castain (rhc_at_[hidden])
Date: 2013-12-21 11:31:36


It seems to be working fine for me:

[rhc_at_bend001 tcp]$ mpirun -np 2 -host bend001 -report-bindings -mca rmaps_lama_bind 1c -mca rmaps lama hostname
bend001
[bend001:17005] MCW rank 1 bound to socket 0[core 1[hwt 0-1]]: [../BB/../../../..][../../../../../..]
[bend001:17005] MCW rank 0 bound to socket 0[core 0[hwt 0-1]]: [BB/../../../../..][../../../../../..]
bend001
[rhc_at_bend001 tcp]$

(I also checked the internals using "-mca rmaps_base_verbose 10") so it could be your hier inversion causing problems again. Or it could be that you are hitting a connection issue we are seeing in some scenarios in the OOB subsystem - though if you are able to run using a non-lama mapper, that would seem unlikely.

On Dec 20, 2013, at 8:09 PM, tmishima_at_[hidden] wrote:

>
>
> Hi Ralph,
>
> Thank you very much. I tried many things such as:
>
> mpirun -np 2 -host node05 -report-bindings -mca rmaps lama -mca
> rmaps_lama_bind 1c myprog
>
> But every try failed. At least they were accepted by openmpi-1.7.3 as far
> as I remember.
> Anyway, please check it when you have a time, because using lama comes from
> my curiosity.
>
> Regards,
> Tetsuya Mishima
>
>
>> I'll try to take a look at it - my expectation is that lama might get
> stuck because you didn't tell it a pattern to map, and I doubt that code
> path has seen much testing.
>>
>>
>> On Dec 20, 2013, at 5:52 PM, tmishima_at_[hidden] wrote:
>>
>>>
>>>
>>> Hi Ralph, I'm glad to hear that, thanks.
>>>
>>> By the way, yesterday I tried to check how lama in 1.7.4rc treat numa
> node.
>>>
>>> Then, even wiht this simple command line, it freezed without any
> massage:
>>>
>>> mpirun -np 2 -host node05 -mca rmaps lama myprog
>>>
>>> Could you check what happened?
>>>
>>> Is it better to open new thread or continue this thread?
>>>
>>> Regards,
>>> Tetsuya Mishima
>>>
>>>
>>>> I'll make it work so that NUMA can be either above or below socket
>>>>
>>>> On Dec 20, 2013, at 2:57 AM, tmishima_at_[hidden] wrote:
>>>>
>>>>>
>>>>>
>>>>> Hi Brice,
>>>>>
>>>>> Thank you for your comment. I understand what you mean.
>>>>>
>>>>> My opinion was made just considering easy way to adjust the code for
>>>>> inversion of hierarchy in object tree.
>>>>>
>>>>> Tetsuya Mishima
>>>>>
>>>>>
>>>>>> I don't think there's any such difference.
>>>>>> Also, all these NUMA architectures are reported the same by hwloc,
> and
>>>>>> therefore used the same in Open MPI.
>>>>>>
>>>>>> And yes, L3 and NUMA are topologically-identical on AMD Magny-Cours
>>> (and
>>>>>> most recent AMD and Intel platforms).
>>>>>>
>>>>>> Brice
>>>>>>
>>>>>>
>>>>>>
>>>>>> Le 20/12/2013 11:33, tmishima_at_[hidden] a écrit :
>>>>>>>
>>>>>>> Hi Ralph,
>>>>>>>
>>>>>>> The numa-node in AMD Mangy-Cours/Interlagos is so called cc(cache
>>>>>>> coherent)NUMA,
>>>>>>> which seems to be a little bit different from the traditional numa
>>>>> defined
>>>>>>> in openmpi.
>>>>>>>
>>>>>>> I notice that ccNUMA object is almost same as L3cache object.
>>>>>>> So "-bind-to l3cache" or "-map-by l3cache" is valid for what I want
>>> to
>>>>> do.
>>>>>>> Therefore, "do not touch it" is one of the solution, I think ...
>>>>>>>
>>>>>>> Anyway, mixing up these two types of numa is the problem.
>>>>>>>
>>>>>>> Regards,
>>>>>>> Tetsuya Mishima
>>>>>>>
>>>>>>>> I can wait it'll be fixed in 1.7.5 or later, because putting
>>> "-bind-to
>>>>>>>> numa"
>>>>>>>> and "-map-by numa" at the same time works as a workaround.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Tetsuya Mishima
>>>>>>>>
>>>>>>>>> Yeah, it will impact everything that uses hwloc topology maps, I
>>>>> fear.
>>>>>>>>>
>>>>>>>>> One side note: you'll need to add --hetero-nodes to your cmd
> line.
>>> If
>>>>>>> we
>>>>>>>> don't see that, we assume that all the node topologies are
> identical
>>> -
>>>>>>>> which clearly isn't true here.
>>>>>>>>> I'll try to resolve the hier inversion over the holiday - won't
> be
>>>>> for
>>>>>>>> 1.7.4, but hopefully for 1.7.5
>>>>>>>>> Thanks
>>>>>>>>> Ralph
>>>>>>>>>
>>>>>>>>> On Dec 18, 2013, at 9:44 PM, tmishima_at_[hidden] wrote:
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> I think it's normal for AMD opteron having 8/16 cores such as
>>>>>>>>>> magny cours or interlagos. Because it usually has 2 numa nodes
>>>>>>>>>> in a cpu(socket), numa-node can not include a socket. This type
>>>>>>>>>> of hierarchy would be natural.
>>>>>>>>>>
>>>>>>>>>> (node03 is Dell PowerEdge R815 and maybe quite common, I guess)
>>>>>>>>>>
>>>>>>>>>> By the way, I think this inversion should affect rmaps_lama
>>> mapping.
>>>>>>>>>>
>>>>>>>>>> Tetsuya Mishima
>>>>>>>>>>
>>>>>>>>>>> Ick - yeah, that would be a problem. I haven't seen that type
> of
>>>>>>>>>> hierarchical inversion before - is node03 a different type of
>>> chip?
>>>>>>>>>>> Might take awhile for me to adjust the code to handle hier
>>>>>>>>>> inversion... :-(
>>>>>>>>>>> On Dec 18, 2013, at 9:05 PM, tmishima_at_[hidden] wrote:
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Hi Ralph,
>>>>>>>>>>>>
>>>>>>>>>>>> I found the reason. I attached the main part of output with 32
>>>>>>>>>>>> core node(node03) and 8 core node(node05) at the bottom.
>>>>>>>>>>>>
>>>>>>>>>>>> From this information, socket of node03 includes numa-node.
>>>>>>>>>>>> On the other hand, numa-node of node05 includes socket.
>>>>>>>>>>>> The direction of object tree is opposite.
>>>>>>>>>>>>
>>>>>>>>>>>> Since "-map-by socket" may be assumed as default,
>>>>>>>>>>>> for node05, "-bind-to numa and -map-by socket" means
>>>>>>>>>>>> upward search. For node03, this should be downward.
>>>>>>>>>>>>
>>>>>>>>>>>> I guess that openmpi-1.7.4rc1 will always assume numa-node
>>>>>>>>>>>> includes socket. Is it right? Then, upward search is assumed
>>>>>>>>>>>> in orte_rmaps_base_compute_bindings even for node03 when I
>>>>>>>>>>>> put "-bind-to numa and -map-by socket" option.
>>>>>>>>>>>>
>>>>>>>>>>>> [node03.cluster:15508] [[38286,0],0] rmaps:base:compute_usage
>>>>>>>>>>>> [node03.cluster:15508] mca:rmaps: compute bindings for job
>>>>>>> [38286,1]
>>>>>>>>>> with
>>>>>>>>>>>> policy NUMA
>>>>>>>>>>>> [node03.cluster:15508] mca:rmaps: bind upwards for job
> [38286,1]
>>>>>>> with
>>>>>>>>>>>> bindings NUMA
>>>>>>>>>>>> [node03.cluster:15508] [[38286,0],0] bind:upward target
> NUMANode
>>>>>>> type
>>>>>>>>>>>> Machine
>>>>>>>>>>>>
>>>>>>>>>>>> That's the reason of this trouble. Therefore, adding "-map-by
>>>>> core"
>>>>>>>>>> works.
>>>>>>>>>>>> (mapping pattern seems to be strange ...)
>>>>>>>>>>>>
>>>>>>>>>>>> [mishima_at_node03 demos]$ mpirun -np 8 -bind-to numa -map-by
> core
>>>>>>>>>>>> -report-bindings myprog
>>>>>>>>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target
> NUMANode
>>>>>>> type
>>>>>>>>>> Cache
>>>>>>>>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target
> NUMANode
>>>>>>> type
>>>>>>>>>> Cache
>>>>>>>>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target
> NUMANode
>>>>>>> type
>>>>>>>>>> Cache
>>>>>>>>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target
> NUMANode
>>>>>>> type
>>>>>>>>>>>> NUMANode
>>>>>>>>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target
> NUMANode
>>>>>>> type
>>>>>>>>>> Cache
>>>>>>>>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target
> NUMANode
>>>>>>> type
>>>>>>>>>> Cache
>>>>>>>>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target
> NUMANode
>>>>>>> type
>>>>>>>>>> Cache
>>>>>>>>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target
> NUMANode
>>>>>>> type
>>>>>>>>>>>> NUMANode
>>>>>>>>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target
> NUMANode
>>>>>>> type
>>>>>>>>>> Cache
>>>>>>>>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target
> NUMANode
>>>>>>> type
>>>>>>>>>> Cache
>>>>>>>>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target
> NUMANode
>>>>>>> type
>>>>>>>>>> Cache
>>>>>>>>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target
> NUMANode
>>>>>>> type
>>>>>>>>>>>> NUMANode
>>>>>>>>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target
> NUMANode
>>>>>>> type
>>>>>>>>>> Cache
>>>>>>>>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target
> NUMANode
>>>>>>> type
>>>>>>>>>> Cache
>>>>>>>>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target
> NUMANode
>>>>>>> type
>>>>>>>>>> Cache
>>>>>>>>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target
> NUMANode
>>>>>>> type
>>>>>>>>>>>> NUMANode
>>>>>>>>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target
> NUMANode
>>>>>>> type
>>>>>>>>>> Cache
>>>>>>>>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target
> NUMANode
>>>>>>> type
>>>>>>>>>> Cache
>>>>>>>>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target
> NUMANode
>>>>>>> type
>>>>>>>>>> Cache
>>>>>>>>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target
> NUMANode
>>>>>>> type
>>>>>>>>>>>> NUMANode
>>>>>>>>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target
> NUMANode
>>>>>>> type
>>>>>>>>>> Cache
>>>>>>>>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target
> NUMANode
>>>>>>> type
>>>>>>>>>> Cache
>>>>>>>>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target
> NUMANode
>>>>>>> type
>>>>>>>>>> Cache
>>>>>>>>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target
> NUMANode
>>>>>>> type
>>>>>>>>>>>> NUMANode
>>>>>>>>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target
> NUMANode
>>>>>>> type
>>>>>>>>>> Cache
>>>>>>>>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target
> NUMANode
>>>>>>> type
>>>>>>>>>> Cache
>>>>>>>>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target
> NUMANode
>>>>>>> type
>>>>>>>>>> Cache
>>>>>>>>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target
> NUMANode
>>>>>>> type
>>>>>>>>>>>> NUMANode
>>>>>>>>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target
> NUMANode
>>>>>>> type
>>>>>>>>>> Cache
>>>>>>>>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target
> NUMANode
>>>>>>> type
>>>>>>>>>> Cache
>>>>>>>>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target
> NUMANode
>>>>>>> type
>>>>>>>>>> Cache
>>>>>>>>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target
> NUMANode
>>>>>>> type
>>>>>>>>>>>> NUMANode
>>>>>>>>>>>> [node03.cluster:15885] MCW rank 2 bound to socket 0[core 0[hwt
>>>>> 0]],
>>>>>>>>>> socket
>>>>>>>>>>>> 0[core 1[hwt 0]], socket 0[core 2[hwt 0]], so
>>>>>>>>>>>> cket 0[core 3[hwt 0]]:
>>>>>>>>>>>>
>>>>>>>
> [B/B/B/B/./././.][./././././././.][./././././././.][./././././././.]
>>>>>>>>>>>> [node03.cluster:15885] MCW rank 3 bound to socket 0[core 0[hwt
>>>>> 0]],
>>>>>>>>>> socket
>>>>>>>>>>>> 0[core 1[hwt 0]], socket 0[core 2[hwt 0]], so
>>>>>>>>>>>> cket 0[core 3[hwt 0]]:
>>>>>>>>>>>>
>>>>>>>
> [B/B/B/B/./././.][./././././././.][./././././././.][./././././././.]
>>>>>>>>>>>> [node03.cluster:15885] MCW rank 4 bound to socket 0[core 4[hwt
>>>>> 0]],
>>>>>>>>>> socket
>>>>>>>>>>>> 0[core 5[hwt 0]], socket 0[core 6[hwt 0]], so
>>>>>>>>>>>> cket 0[core 7[hwt 0]]:
>>>>>>>>>>>>
>>>>>>>
> [././././B/B/B/B][./././././././.][./././././././.][./././././././.]
>>>>>>>>>>>> [node03.cluster:15885] MCW rank 5 bound to socket 0[core 4[hwt
>>>>> 0]],
>>>>>>>>>> socket
>>>>>>>>>>>> 0[core 5[hwt 0]], socket 0[core 6[hwt 0]], so
>>>>>>>>>>>> cket 0[core 7[hwt 0]]:
>>>>>>>>>>>>
>>>>>>>
> [././././B/B/B/B][./././././././.][./././././././.][./././././././.]
>>>>>>>>>>>> [node03.cluster:15885] MCW rank 6 bound to socket 0[core 4[hwt
>>>>> 0]],
>>>>>>>>>> socket
>>>>>>>>>>>> 0[core 5[hwt 0]], socket 0[core 6[hwt 0]], so
>>>>>>>>>>>> cket 0[core 7[hwt 0]]:
>>>>>>>>>>>>
>>>>>>>
> [././././B/B/B/B][./././././././.][./././././././.][./././././././.]
>>>>>>>>>>>> [node03.cluster:15885] MCW rank 7 bound to socket 0[core 4[hwt
>>>>> 0]],
>>>>>>>>>> socket
>>>>>>>>>>>> 0[core 5[hwt 0]], socket 0[core 6[hwt 0]], so
>>>>>>>>>>>> cket 0[core 7[hwt 0]]:
>>>>>>>>>>>>
>>>>>>>
> [././././B/B/B/B][./././././././.][./././././././.][./././././././.]
>>>>>>>>>>>> [node03.cluster:15885] MCW rank 0 bound to socket 0[core 0[hwt
>>>>> 0]],
>>>>>>>>>> socket
>>>>>>>>>>>> 0[core 1[hwt 0]], socket 0[core 2[hwt 0]], so
>>>>>>>>>>>> cket 0[core 3[hwt 0]]:
>>>>>>>>>>>>
>>>>>>>
> [B/B/B/B/./././.][./././././././.][./././././././.][./././././././.]
>>>>>>>>>>>> [node03.cluster:15885] MCW rank 1 bound to socket 0[core 0[hwt
>>>>> 0]],
>>>>>>>>>> socket
>>>>>>>>>>>> 0[core 1[hwt 0]], socket 0[core 2[hwt 0]], so
>>>>>>>>>>>> cket 0[core 3[hwt 0]]:
>>>>>>>>>>>>
>>>>>>>
> [B/B/B/B/./././.][./././././././.][./././././././.][./././././././.]
>>>>>>>>>>>> Hello world from process 6 of 8
>>>>>>>>>>>> Hello world from process 5 of 8
>>>>>>>>>>>> Hello world from process 0 of 8
>>>>>>>>>>>> Hello world from process 7 of 8
>>>>>>>>>>>> Hello world from process 3 of 8
>>>>>>>>>>>> Hello world from process 4 of 8
>>>>>>>>>>>> Hello world from process 2 of 8
>>>>>>>>>>>> Hello world from process 1 of 8
>>>>>>>>>>>>
>>>>>>>>>>>> Regards,
>>>>>>>>>>>> Tetsuya Mishima
>>>>>>>>>>>>
>>>>>>>>>>>> [node03.cluster:15508] Type: Machine Number of child objects:
> 4
>>>>>>>>>>>> Name=NULL
>>>>>>>>>>>> total=132358820KB
>>>>>>>>>>>> Backend=Linux
>>>>>>>>>>>> OSName=Linux
>>>>>>>>>>>> OSRelease=2.6.18-308.16.1.el5
>>>>>>>>>>>> OSVersion="#1 SMP Tue Oct 2 22:01:43 EDT 2012"
>>>>>>>>>>>> Architecture=x86_64
>>>>>>>>>>>> Cpuset: 0xffffffff
>>>>>>>>>>>> Online: 0xffffffff
>>>>>>>>>>>> Allowed: 0xffffffff
>>>>>>>>>>>> Bind CPU proc: TRUE
>>>>>>>>>>>> Bind CPU thread: TRUE
>>>>>>>>>>>> Bind MEM proc: FALSE
>>>>>>>>>>>> Bind MEM thread: TRUE
>>>>>>>>>>>> Type: Socket Number of child objects: 2
>>>>>>>>>>>> Name=NULL
>>>>>>>>>>>> total=33071780KB
>>>>>>>>>>>> CPUModel="AMD Opteron(tm) Processor 6136"
>>>>>>>>>>>> Cpuset: 0x000000ff
>>>>>>>>>>>> Online: 0x000000ff
>>>>>>>>>>>> Allowed: 0x000000ff
>>>>>>>>>>>> Type: NUMANode Number of child objects: 1
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> [node05.cluster:21750] Type: Machine Number of child objects:
> 2
>>>>>>>>>>>> Name=NULL
>>>>>>>>>>>> total=33080072KB
>>>>>>>>>>>> Backend=Linux
>>>>>>>>>>>> OSName=Linux
>>>>>>>>>>>> OSRelease=2.6.18-308.16.1.el5
>>>>>>>>>>>> OSVersion="#1 SMP Tue Oct 2 22:01:43 EDT 2012"
>>>>>>>>>>>> Architecture=x86_64
>>>>>>>>>>>> Cpuset: 0x000000ff
>>>>>>>>>>>> Online: 0x000000ff
>>>>>>>>>>>> Allowed: 0x000000ff
>>>>>>>>>>>> Bind CPU proc: TRUE
>>>>>>>>>>>> Bind CPU thread: TRUE
>>>>>>>>>>>> Bind MEM proc: FALSE
>>>>>>>>>>>> Bind MEM thread: TRUE
>>>>>>>>>>>> Type: NUMANode Number of child objects: 1
>>>>>>>>>>>> Name=NULL
>>>>>>>>>>>> local=16532232KB
>>>>>>>>>>>> total=16532232KB
>>>>>>>>>>>> Cpuset: 0x0000000f
>>>>>>>>>>>> Online: 0x0000000f
>>>>>>>>>>>> Allowed: 0x0000000f
>>>>>>>>>>>> Type: Socket Number of child objects: 1
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>> Hmm...try adding "-mca rmaps_base_verbose 10 -mca
>>>>> ess_base_verbose
>>>>>>>> 5"
>>>>>>>>>> to
>>>>>>>>>>>> your cmd line and let's see what it thinks it found.
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Dec 18, 2013, at 6:55 PM, tmishima_at_[hidden]
> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi, I report one more problem with openmpi-1.7.4rc1,
>>>>>>>>>>>>>> which is more serious.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> For our 32 core nodes(AMD magny cours based) which has
>>>>>>>>>>>>>> 8 numa-nodes, "-bind-to numa" does not work. Without
>>>>>>>>>>>>>> this option, it works. For your infomation, at the
>>>>>>>>>>>>>> bottom of this mail, I added the lstopo information
>>>>>>>>>>>>>> of the node.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>> Tetsuya Mishima
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> [mishima_at_manage ~]$ qsub -I -l nodes=1:ppn=32
>>>>>>>>>>>>>> qsub: waiting for job 8352.manage.cluster to start
>>>>>>>>>>>>>> qsub: job 8352.manage.cluster ready
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> [mishima_at_node03 demos]$ mpirun -np 8 -report-bindings
> -bind-to
>>>>>>> numa
>>>>>>>>>>>> myprog
>>>>>>>>>>>>>> [node03.cluster:15316] [[37582,0],0] bind:upward target
>>> NUMANode
>>>>>>>> type
>>>>>>>>>>>>>> Machine
>>>>>>>>>>>>>>
>>>>>>>
>>>>>
>>>
> --------------------------------------------------------------------------
>>>>>>>>>>>>>> A request was made to bind to NUMA, but an appropriate
> target
>>>>>>> could
>>>>>>>>>> not
>>>>>>>>>>>>>> be found on node node03.
>>>>>>>>>>>>>>
>>>>>>>
>>>>>
>>>
> --------------------------------------------------------------------------
>>>>>>>>>>>>>> [mishima_at_node03 ~]$ cd ~/Desktop/openmpi-1.7/demos/
>>>>>>>>>>>>>> [mishima_at_node03 demos]$ mpirun -np 8 -report-bindings myprog
>>>>>>>>>>>>>> [node03.cluster:15282] MCW rank 2 bound to socket 1[core 8
> [hwt
>>>>>>> 0]]:
>>>>>>>>>>>>>> [./././././././.][B/././././././.][./././././././.][
>>>>>>>>>>>>>> ./././././././.]>>>>>>>>>>>> [node03.cluster:15282] MCW rank
> 3 bound to socket 1[core 9[hwt
>>>>>>> 0]]:
>>>>>>>>>>>>>> [./././././././.][./B/./././././.][./././././././.][
>>>>>>>>>>>>>> ./././././././.]
>>>>>>>>>>>>>> [node03.cluster:15282] MCW rank 4 bound to socket 2[core 16
>>> [hwt
>>>>>>>> 0]]:
>>>>>>>>>>>>>> [./././././././.][./././././././.][B/././././././.]
>>>>>>>>>>>>>> [./././././././.]
>>>>>>>>>>>>>> [node03.cluster:15282] MCW rank 5 bound to socket 2[core 17
>>> [hwt
>>>>>>>> 0]]:
>>>>>>>>>>>>>> [./././././././.][./././././././.][./B/./././././.]
>>>>>>>>>>>>>> [./././././././.]
>>>>>>>>>>>>>> [node03.cluster:15282] MCW rank 6 bound to socket 3[core 24
>>> [hwt
>>>>>>>> 0]]:
>>>>>>>>>>>>>> [./././././././.][./././././././.][./././././././.]
>>>>>>>>>>>>>> [B/././././././.]
>>>>>>>>>>>>>> [node03.cluster:15282] MCW rank 7 bound to socket 3[core 25
>>> [hwt
>>>>>>>> 0]]:
>>>>>>>>>>>>>> [./././././././.][./././././././.][./././././././.]
>>>>>>>>>>>>>> [./B/./././././.]
>>>>>>>>>>>>>> [node03.cluster:15282] MCW rank 0 bound to socket 0[core 0
> [hwt
>>>>>>> 0]]:
>>>>>>>>>>>>>> [B/././././././.][./././././././.][./././././././.][
>>>>>>>>>>>>>> ./././././././.]
>>>>>>>>>>>>>> [node03.cluster:15282] MCW rank 1 bound to socket 0[core 1
> [hwt
>>>>>>> 0]]:
>>>>>>>>>>>>>> [./B/./././././.][./././././././.][./././././././.][
>>>>>>>>>>>>>> ./././././././.]
>>>>>>>>>>>>>> Hello world from process 2 of 8
>>>>>>>>>>>>>> Hello world from process 5 of 8
>>>>>>>>>>>>>> Hello world from process 4 of 8
>>>>>>>>>>>>>> Hello world from process 3 of 8>>>>>>>>>> Hello world from
>>> process 1 of 8
>>>>>>>>>>>>>> Hello world from process 7 of 8
>>>>>>>>>>>>>> Hello world from process 6 of 8
>>>>>>>>>>>>>> Hello world from process 0 of 8
>>>>>>>>>>>>>> [mishima_at_node03 demos]$ ~/opt/hwloc/bin/lstopo-no-graphics
>>>>>>>>>>>>>> Machine (126GB)
>>>>>>>>>>>>>> Socket L#0 (32GB)
>>>>>>>>>>>>>> NUMANode L#0 (P#0 16GB) + L3 L#0 (5118KB)
>>>>>>>>>>>>>> L2 L#0 (512KB) + L1d L#0 (64KB) + L1i L#0 (64KB) + Core L#0
>>> +
>>>>>>> PU
>>>>>>>>>>>> L#0
>>>>>>>>>>>>>> (P#0)
>>>>>>>>>>>>>> L2 L#1 (512KB) + L1d L#1 (64KB) + L1i L#1 (64KB) + Core L#1
>>> +
>>>>>>> PU
>>>>>>>>>>>> L#1
>>>>>>>>>>>>>> (P#1)
>>>>>>>>>>>>>> L2 L#2 (512KB) + L1d L#2 (64KB) + L1i L#2 (64KB) + Core L#2
>>> +
>>>>>>> PU
>>>>>>>>>>>> L#2
>>>>>>>>>>>>>> (P#2)
>>>>>>>>>>>>>> L2 L#3 (512KB) + L1d L#3 (64KB) + L1i L#3 (64KB) + Core L#3
>>> +
>>>>>>> PU
>>>>>>>>>>>> L#3
>>>>>>>>>>>>>> (P#3)
>>>>>>>>>>>>>> NUMANode L#1 (P#1 16GB) + L3 L#1 (5118KB)
>>>>>>>>>>>>>> L2 L#4 (512KB) + L1d L#4 (64KB) + L1i L#4 (64KB) + Core L#4
>>> +
>>>>>>> PU
>>>>>>>>>>>> L#4
>>>>>>>>>>>>>> (P#4)
>>>>>>>>>>>>>> L2 L#5 (512KB) + L1d L#5 (64KB) + L1i L#5 (64KB) + Core L#5
>>> +
>>>>>>> PU
>>>>>>>>>>>> L#5
>>>>>>>>>>>>>> (P#5)
>>>>>>>>>>>>>> L2 L#6 (512KB) + L1d L#6 (64KB) + L1i L#6 (64KB) + Core L#6
>>> +
>>>>>>> PU
>>>>>>>>>>>> L#6
>>>>>>>>>>>>>> (P#6)
>>>>>>>>>>>>>> L2 L#7 (512KB) + L1d L#7 (64KB) + L1i L#7 (64KB) + Core L#7
>>> +
>>>>>>> PU>>>>>> L#7
>>>>>>>>>>>>>> (P#7)
>>>>>>>>>>>>>> Socket L#1 (32GB)
>>>>>>>>>>>>>> NUMANode L#2 (P#6 16GB) + L3 L#2 (5118KB)
>>>>>>>>>>>>>> L2 L#8 (512KB) + L1d L#8 (64KB) + L1i L#8 (64KB) + Core L#8
>>> +
>>>>>>> PU
>>>>>>>>>>>> L#8
>>>>>>>>>>>>>> (P#8)
>>>>>>>>>>>>>> L2 L#9 (512KB) + L1d L#9 (64KB) + L1i L#9 (64KB) + Core L#9
>>> +
>>>>>>> PU
>>>>>>>>>>>> L#9
>>>>>>>>>>>>>> (P#9)
>>>>>>>>>>>>>> L2 L#10 (512KB) + L1d L#10 (64KB) + L1i L#10 (64KB) + Core
>>>>>>> L#10
>>>>>>>> +
>>>>>>>>>>>> PU
>>>>>>>>>>>>>> L#10 (P#10)
>>>>>>>>>>>>>> L2 L#11 (512KB) + L1d L#11 (64KB) + L1i L#11 (64KB) + Core
>>>>>>> L#11
>>>>>>>> +
>>>>>>>>>>>> PU
>>>>>>>>>>>>>> L#11 (P#11)
>>>>>>>>>>>>>> NUMANode L#3 (P#7 16GB) + L3 L#3 (5118KB)
>>>>>>>>>>>>>> L2 L#12 (512KB) + L1d L#12 (64KB) + L1i L#12 (64KB) + Core
>>>>>>> L#12
>>>>>>>> +
>>>>>>>>>>>> PU
>>>>>>>>>>>>>> L#12 (P#12)
>>>>>>>>>>>>>> L2 L#13 (512KB) + L1d L#13 (64KB) + L1i L#13 (64KB) + Core
>>>>>>> L#13
>>>>>>>> +
>>>>>>>>>>>> PU
>>>>>>>>>>>>>> L#13 (P#13)
>>>>>>>>>>>>>> L2 L#14 (512KB) + L1d L#14 (64KB) + L1i L#14 (64KB) + Core
>>>>>>> L#14
>>>>>>>> +
>>>>>>>>>>>> PU
>>>>>>>>>>>>>> L#14 (P#14)
>>>>>>>>>>>>>> L2 L#15 (512KB) + L1d L#15 (64KB) + L1i L#15 (64KB) + Core
>>>>>>> L#15
>>>>>>>> +
>>>>>>>>>>>> PU
>>>>>>>>>>>>>> L#15 (P#15)
>>>>>>>>>>>>>> Socket L#2 (32GB)
>>>>>>>>>>>>>> NUMANode L#4 (P#4 16GB) + L3 L#4 (5118KB)
>>>>>>>>>>>>>> L2 L#16 (512KB) + L1d L#16 (64KB) + L1i L#16 (64KB) + Core
>>>>>>> L#16
>>>>>>>> +
>>>>>>>>>>>> PU
>>>>>>>>>>>>>> L#16 (P#16)
>>>>>>>>>>>>>> L2 L#17 (512KB) + L1d L#17 (64KB) + L1i L#17 (64KB) + Core
>>>>>>> L#17
>>>>>>>> +
>>>>>>>>>>>> PU
>>>>>>>>>>>>>> L#17 (P#17)> >>>>> L2 L#18 (512KB) + L1d L#18 (64KB) +
> L1i
>>>>>>> L#18 (64KB) + Core L#18
>>>>>>>> +
>>>>>>>>>>>> PU
>>>>>>>>>>>>>> L#18 (P#18)
>>>>>>>>>>>>>> L2 L#19 (512KB) + L1d L#19 (64KB) + L1i L#19 (64KB) + Core
>>>>>>> L#19
>>>>>>>> +
>>>>>>>>>>>> PU
>>>>>>>>>>>>>> L#19 (P#19)
>>>>>>>>>>>>>> NUMANode L#5 (P#5 16GB) + L3 L#5 (5118KB)
>>>>>>>>>>>>>> L2 L#20 (512KB) + L1d L#20 (64KB) + L1i L#20 (64KB) + Core
>>>>>>> L#20
>>>>>>>> +
>>>>>>>>>>>> PU
>>>>>>>>>>>>>> L#20 (P#20)
>>>>>>>>>>>>>> L2 L#21 (512KB) + L1d L#21 (64KB) + L1i L#21 (64KB) + Core
>>>>>>> L#21
>>>>>>>> +
>>>>>>>>>>>> PU
>>>>>>>>>>>>>> L#21 (P#21)
>>>>>>>>>>>>>> L2 L#22 (512KB) + L1d L#22 (64KB) + L1i L#22 (64KB) + Core
>>>>>>> L#22
>>>>>>>> +
>>>>>>>>>>>> PU
>>>>>>>>>>>>>> L#22 (P#22)
>>>>>>>>>>>>>> L2 L#23 (512KB) + L1d L#23 (64KB) + L1i L#23 (64KB) + Core
>>>>>>> L#23
>>>>>>>> +
>>>>>>>>>>>> PU
>>>>>>>>>>>>>> L#23 (P#23)
>>>>>>>>>>>>>> Socket L#3 (32GB)
>>>>>>>>>>>>>> NUMANode L#6 (P#2 16GB) + L3 L#6 (5118KB)
>>>>>>>>>>>>>> L2 L#24 (512KB) + L1d L#24 (64KB) + L1i L#24 (64KB) + Core
>>>>>>> L#24
>>>>>>>> +
>>>>>>>>>>>> PU
>>>>>>>>>>>>>> L#24 (P#24)>>>>> L2 L#25 (512KB) + L1d L#25 (64KB) + L1i
>>> L#25
>>>>>>>> (64KB) + Core L#25 +
>>>>>>>>>>>> PU
>>>>>>>>>>>>>> L#25 (P#25)
>>>>>>>>>>>>>> L2 L#26 (512KB) + L1d L#26 (64KB) + L1i L#26 (64KB) + Core
>>>>>>> L#26
>>>>>>>> +
>>>>>>>>>>>> PU
>>>>>>>>>>>>>> L#26 (P#26)
>>>>>>>>>>>>>> L2 L#27 (512KB) + L1d L#27 (64KB) + L1i L#27 (64KB) + Core
>>>>>>> L#27
>>>>>>>> +
>>>>>>>>>>>> PU
>>>>>>>>>>>>>> L#27 (P#27)
>>>>>>>>>>>>>> NUMANode L#7 (P#3 16GB) + L3 L#7 (5118KB)
>>>>>>>>>>>>>> L2 L#28 (512KB) + L1d L#28 (64KB) + L1i L#28 (64KB) + Core
>>>>>>> L#28
>>>>>>>> +
>>>>>>>>>>>> PU
>>>>>>>>>>>>>> L#28 (P#28)
>>>>>>>>>>>>>> L2 L#29 (512KB) + L1d L#29 (64KB) + L1i L#29 (64KB) + Core
>>>>>>> L#29
>>>>>>>> +
>>>>>>>>>>>> PU
>>>>>>>>>>>>>> L#29 (P#29)
>>>>>>>>>>>>>> L2 L#30 (512KB) + L1d L#30 (64KB) + L1i L#30 (64KB) + Core
>>>>>>> L#30
>>>>>>>> +
>>>>>>>>>>>> PU
>>>>>>>>>>>>>> L#30 (P#30)
>>>>>>>>>>>>>> L2 L#31 (512KB) + L1d L#31 (64KB) + L1i L#31 (64KB) + Core
>>>>>>> L#31
>>>>>>>> +
>>>>>>>>>>>> PU
>>>>>>>>>>>>>> L#31 (P#31)
>>>>>>>>>>>>>> HostBridge L#0
>>>>>>>>>>>>>> PCIBridge
>>>>>>>>>>>>>> PCI 14e4:1639
>>>>>>>>>>>>>> Net L#0 "eth0"
>>>>>>>>>>>>>> PCI 14e4:1639
>>>>>>>>>>>>>> Net L#1 "eth1"
>>>>>>>>>>>>>> PCIBridge
>>>>>>>>>>>>>> PCI 14e4:1639
>>>>>>>>>>>>>> Net L#2 "eth2"
>>>>>>>>>>>>>> PCI 14e4:1639
>>>>>>>>>>>>>> Net L#3 "eth3"
>>>>>>>>>>>>>> PCIBridge
>>>>>>>>>>>>>> PCIBridge
>>>>>>>>>>>>>> PCIBridge
>>>>>>>>>>>>>> PCI 1000:0072
>>>>>>>>>>>>>> Block L#4 "sdb"
>>>>>>>>>>>>>> Block L#5 "sda"
>>>>>>>>>>>>>> PCI 1002:4390
>>>>>>>>>>>>>> Block L#6 "sr0"
>>>>>>>>>>>>>> PCIBridge
>>>>>>>>>>>>>> PCI 102b:0532
>>>>>>>>>>>>>> HostBridge L#7
>>>>>>>>>>>>>> PCIBridge
>>>>>>>>>>>>>> PCI 15b3:6274
>>>>>>>>>>>>>> Net L#7 "ib0"
>>>>>>>>>>>>>> OpenFabrics L#8 "mthca0"
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>>> users mailing list
>>>>>>>>>>>>>> users_at_[hidden]
>>>>>>>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>> users mailing list
>>>>>>>>>>>>> users_at_[hidden]>>
>>>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>> users mailing list
>>>>>>>>>>>> users_at_[hidden]
>>>>>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>>>>>>>> _______________________________________________
>>>>>>>>>>> users mailing list
>>>>>>>>>>> users_at_[hidden]
>>>>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>>>>>>> _______________________________________________
>>>>>>>>>> users mailing list
>>>>>>>>>> users_at_[hidden]
>>>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>>>>>> _______________________________________________
>>>>>>>>> users mailing list
>>>>>>>>> users_at_[hidden]
>>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>>>>> _______________________________________________
>>>>>>>> users mailing list
>>>>>>>> users_at_[hidden]
>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>>>> _______________________________________________
>>>>>>> users mailing list
>>>>>>> users_at_[hidden]
>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>>>
>>>>>> _______________________________________________
>>>>>> users mailing list
>>>>>> users_at_[hidden]
>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>>
>>>>> _______________________________________________
>>>>> users mailing list
>>>>> users_at_[hidden]
>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>
>>>> _______________________________________________
>>>> users mailing list
>>>> users_at_[hidden]
>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>
>>> _______________________________________________
>>> users mailing list
>>> users_at_[hidden]
>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>
>> _______________________________________________
>> users mailing list
>> users_at_[hidden]
>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>
> _______________________________________________
> users mailing list
> users_at_[hidden]
> http://www.open-mpi.org/mailman/listinfo.cgi/users