Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: Re: [OMPI users] "-bind-to numa" of openmpi-1.7.4rc1 dosen't work for our magny cours based 32 core node
From: Ralph Castain (rhc_at_[hidden])
Date: 2013-12-22 00:44:01


On Dec 21, 2013, at 8:16 PM, tmishima_at_[hidden] wrote:

>
>
> Ralph, thanks. I'll try it on Tuseday.
>
> Let me confirm one thing. I don't put "-with-libevent" when I build
> openmpi.
> Is there any possibility to build with external libevent automatically?

No - only happens if you direct it

>
> Tetsuya Mishima
>
>
>> Not entirely sure - add "-mca rmaps_base_verbose 10 --display-map" to
> your cmd line and let's see if it finishes the mapping.
>>
>> Unless you specifically built with an external libevent (which I doubt),
> there is no conflict. The connection issue is unlikely to be a factor here
> as it works when not using the lama mapper.
>>
>>
>> On Dec 21, 2013, at 3:43 PM, tmishima_at_[hidden] wrote:
>>
>>>
>>>
>>> Thank you, Ralph.
>>>
>>> Then, this problem should depend on our environment.
>>> But, at least, inversion problem is not the cause because
>>> node05 has normal hier order.
>>>
>>> I can not connect to our cluster now. Tuesday, going
>>> back to my office, I'll send you further report.
>>>
>>> Before that, please let me know your configuration. I will
>>> follow your configuation as much as possible. Our configuraion
>>> is very simple, only -with-tm -with-ibverbs -disable-ipv6.
>>> (on CentOS 5.8)
>>>
>>> The 1.7 series is a llite bit unstable on our cluster yet.
>>>
>>> Similar freezing(hang up) was observed with 1.7.3. At that
>>> time, lama worked well but putting "-rank-by something" caused
>>> same freezing (curiously, rank-by works with 1.7.4rc1).
>>> I checked where it stopped using gdb, then I found that it
>>> stopped to wait for event in a function of libevent(I can not
>>> recall the name).
>>>
>>> Is this related to your "connection issue in the OOB
>>> subsystem"? Or libevent version conflict? I guess these two
>>> problems are related each other. They stopped at very early
>>> stage before reaching mapping function because no message
>>> appeared before freezing, which is my random guess.
>>>
>>> Could you give me any hint or comment?
>>>
>>> Regards,
>>> Tetsuya Mishima
>>>
>>>
>>>> It seems to be working fine for me:
>>>>
>>>> [rhc_at_bend001 tcp]$ mpirun -np 2 -host bend001 -report-bindings -mca
>>> rmaps_lama_bind 1c -mca rmaps lama hostname
>>>> bend001
>>>> [bend001:17005] MCW rank 1 bound to socket 0[core 1[hwt 0-1]]:
>>> [../BB/../../../..][../../../../../..]
>>>> [bend001:17005] MCW rank 0 bound to socket 0[core 0[hwt 0-1]]:
>>> [BB/../../../../..][../../../../../..]
>>>> bend001
>>>> [rhc_at_bend001 tcp]$
>>>>
>>>> (I also checked the internals using "-mca rmaps_base_verbose 10") so
> it
>>> could be your hier inversion causing problems again. Or it could be
> that
>>> you are hitting a connection issue we are seeing in
>>>> some scenarios in the OOB subsystem - though if you are able to run
> using
>>> a non-lama mapper, that would seem unlikely.
>>>>
>>>>
>>>> On Dec 20, 2013, at 8:09 PM, tmishima_at_[hidden] wrote:
>>>>
>>>>
>>>>
>>>> Hi Ralph,
>>>>
>>>> Thank you very much. I tried many things such as:
>>>>
>>>> mpirun -np 2 -host node05 -report-bindings -mca rmaps lama -mca
>>>> rmaps_lama_bind 1c myprog
>>>>
>>>> But every try failed. At least they were accepted by openmpi-1.7.3 as
> far
>>>> as I remember.
>>>> Anyway, please check it when you have a time, because using lama comes
>>> from
>>>> my curiosity.
>>>>
>>>> Regards,
>>>> Tetsuya Mishima
>>>>
>>>>
>>>> I'll try to take a look at it - my expectation is that lama might get
>>>> stuck because you didn't tell it a pattern to map, and I doubt that
> code
>>>> path has seen much testing.
>>>>
>>>>
>>>> On Dec 20, 2013, at 5:52 PM, tmishima_at_[hidden] wrote:
>>>>
>>>>
>>>>
>>>> Hi Ralph, I'm glad to hear that, thanks.
>>>>
>>>> By the way, yesterday I tried to check how lama in 1.7.4rc treat numa
>>>> node.
>>>>
>>>> Then, even wiht this simple command line, it freezed without any
>>>> massage:
>>>>
>>>> mpirun -np 2 -host node05 -mca rmaps lama myprog
>>>>
>>>> Could you check what happened?
>>>>
>>>> Is it better to open new thread or continue this thread?
>>>>
>>>> Regards,
>>>> Tetsuya Mishima
>>>>
>>>>
>>>> I'll make it work so that NUMA can be either above or below socket
>>>>
>>>> On Dec 20, 2013, at 2:57 AM, tmishima_at_[hidden] wrote:
>>>>
>>>>
>>>>
>>>> Hi Brice,
>>>>
>>>> Thank you for your comment. I understand what you mean.
>>>>
>>>> My opinion was made just considering easy way to adjust the code for
>>>> inversion of hierarchy in object tree.
>>>>
>>>> Tetsuya Mishima
>>>>
>>>>
>>>> I don't think there's any such difference.
>>>> Also, all these NUMA architectures are reported the same by hwloc,
>>>> and
>>>> therefore used the same in Open MPI.
>>>>
>>>> And yes, L3 and NUMA are topologically-identical on AMD Magny-Cours
>>>> (and
>>>> most recent AMD and Intel platforms).
>>>>
>>>> Brice
>>>>
>>>>
>>>>
>>>> Le 20/12/2013 11:33, tmishima_at_[hidden] a écrit :
>>>>
>>>> Hi Ralph,
>>>>
>>>> The numa-node in AMD Mangy-Cours/Interlagos is so called cc(cache
>>>> coherent)NUMA,
>>>> which seems to be a little bit different from the traditional numa
>>>> defined
>>>> in openmpi.
>>>>
>>>> I notice that ccNUMA object is almost same as L3cache object.
>>>> So "-bind-to l3cache" or "-map-by l3cache" is valid for what I want
>>>> to
>>>> do.
>>>> Therefore, "do not touch it" is one of the solution, I think ...
>>>>
>>>> Anyway, mixing up these two types of numa is the problem.
>>>>
>>>> Regards,
>>>> Tetsuya Mishima
>>>>
>>>> I can wait it'll be fixed in 1.7.5 or later, because putting
>>>> "-bind-to
>>>> numa"
>>>> and "-map-by numa" at the same time works as a workaround.
>>>>
>>>> Thanks,
>>>> Tetsuya Mishima
>>>>
>>>> Yeah, it will impact everything that uses hwloc topology maps, I
>>>> fear.
>>>>
>>>> One side note: you'll need to add --hetero-nodes to your cmd
>>>> line.
>>>> If
>>>> we
>>>> don't see that, we assume that all the node topologies are
>>>> identical
>>>> -
>>>> which clearly isn't true here.
>>>> I'll try to resolve the hier inversion over the holiday - won't
>>>> be
>>>> for
>>>> 1.7.4, but hopefully for 1.7.5
>>>> Thanks
>>>> Ralph
>>>>
>>>> On Dec 18, 2013, at 9:44 PM, tmishima_at_[hidden] wrote:
>>>>
>>>>
>>>> I think it's normal for AMD opteron having 8/16 cores such as
>>>> magny cours or interlagos. Because it usually has 2 numa nodes
>>>> in a cpu(socket), numa-node can not include a socket. This type
>>>> of hierarchy would be natural.
>>>>
>>>> (node03 is Dell PowerEdge R815 and maybe quite common, I guess)
>>>>
>>>> By the way, I think this inversion should affect rmaps_lama
>>>> mapping.
>>>>
>>>> Tetsuya Mishima
>>>>
>>>> Ick - yeah, that would be a problem. I haven't seen that type
>>>> of
>>>> hierarchical inversion before - is node03 a different type of
>>>> chip?
>>>> Might take awhile for me to adjust the code to handle hier
>>>> inversion... :-(
>>>> On Dec 18, 2013, at 9:05 PM, tmishima_at_[hidden] wrote:
>>>>
>>>>
>>>> Hi Ralph,
>>>>
>>>> I found the reason. I attached the main part of output with 32
>>>> core node(node03) and 8 core node(node05) at the bottom.
>>>>
>>>> From this information, socket of node03 includes numa-node.
>>>> On the other hand, numa-node of node05 includes socket.
>>>> The direction of object tree is opposite.
>>>>
>>>> Since "-map-by socket" may be assumed as default,
>>>> for node05, "-bind-to numa and -map-by socket" means
>>>> upward search. For node03, this should be downward.
>>>>
>>>> I guess that openmpi-1.7.4rc1 will always assume numa-node
>>>> includes socket. Is it right? Then, upward search is assumed
>>>> in orte_rmaps_base_compute_bindings even for node03 when I
>>>> put "-bind-to numa and -map-by socket" option.
>>>>
>>>> [node03.cluster:15508] [[38286,0],0] rmaps:base:compute_usage
>>>> [node03.cluster:15508] mca:rmaps: compute bindings for job
>>>> [38286,1]
>>>> with
>>>> policy NUMA
>>>> [node03.cluster:15508] mca:rmaps: bind upwards for job
>>>> [38286,1]
>>>> with
>>>> bindings NUMA
>>>> [node03.cluster:15508] [[38286,0],0] bind:upward target
>>>> NUMANode
>>>> type
>>>> Machine
>>>>
>>>> That's the reason of this trouble. Therefore, adding "-map-by
>>>> core"
>>>> works.
>>>> (mapping pattern seems to be strange ...)
>>>>
>>>> [mishima_at_node03 demos]$ mpirun -np 8 -bind-to numa -map-by
>>>> core
>>>> -report-bindings myprog
>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target
>>>> NUMANode
>>>> type
>>>> Cache
>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target
>>>> NUMANode
>>>> type
>>>> Cache
>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target
>>>> NUMANode
>>>> type
>>>> Cache
>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target
>>>> NUMANode
>>>> type
>>>> NUMANode
>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target
>>>> NUMANode
>>>> type
>>>> Cache
>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target
>>>> NUMANode
>>>> type
>>>> Cache
>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target
>>>> NUMANode
>>>> type
>>>> Cache
>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target
>>>> NUMANode
>>>> type
>>>> NUMANode
>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target
>>>> NUMANode
>>>> type
>>>> Cache
>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target
>>>> NUMANode
>>>> type
>>>> Cache
>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target
>>>> NUMANode
>>>> type
>>>> Cache
>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target
>>>> NUMANode
>>>> type
>>>> NUMANode
>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target
>>>> NUMANode
>>>> type
>>>> Cache
>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target
>>>> NUMANode
>>>> type
>>>> Cache
>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target
>>>> NUMANode
>>>> type
>>>> Cache
>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target
>>>> NUMANode
>>>> type
>>>> NUMANode
>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target
>>>> NUMANode
>>>> type
>>>> Cache
>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target
>>>> NUMANode
>>>> type
>>>> Cache
>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target
>>>> NUMANode
>>>> type
>>>> Cache
>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target
>>>> NUMANode
>>>> type
>>>> NUMANode
>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target
>>>> NUMANode
>>>> type
>>>> Cache
>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target
>>>> NUMANode
>>>> type
>>>> Cache
>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target
>>>> NUMANode
>>>> type
>>>> Cache
>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target
>>>> NUMANode
>>>> type
>>>> NUMANode
>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target
>>>> NUMANode
>>>> type
>>>> Cache
>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target
>>>> NUMANode
>>>> type
>>>> Cache
>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target
>>>> NUMANode
>>>> type
>>>> Cache
>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target
>>>> NUMANode
>>>> type
>>>> NUMANode
>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target
>>>> NUMANode
>>>> type
>>>> Cache
>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target
>>>> NUMANode
>>>> type
>>>> Cache
>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target
>>>> NUMANode
>>>> type
>>>> Cache
>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target
>>>> NUMANode
>>>> type
>>>> NUMANode
>>>> [node03.cluster:15885] MCW rank 2 bound to socket 0[core 0[hwt
>>>> 0]],
>>>> socket
>>>> 0[core 1[hwt 0]], socket 0[core 2[hwt 0]], so
>>>> cket 0[core 3[hwt 0]]:
>>>>
>>>>
>>>> [B/B/B/B/./././.][./././././././.][./././././././.][./././././././.]
>>>> [node03.cluster:15885] MCW rank 3 bound to socket 0[core 0[hwt
>>>> 0]],
>>>> socket
>>>> 0[core 1[hwt 0]], socket 0[core 2[hwt 0]], so
>>>> cket 0[core 3[hwt 0]]:
>>>>
>>>>
>>>> [B/B/B/B/./././.][./././././././.][./././././././.][./././././././.]
>>>> [node03.cluster:15885] MCW rank 4 bound to socket 0[core 4[hwt
>>>> 0]],
>>>> socket
>>>> 0[core 5[hwt 0]], socket 0[core 6[hwt 0]], so
>>>> cket 0[core 7[hwt 0]]:
>>>>
>>>>
>>>> [././././B/B/B/B][./././././././.][./././././././.][./././././././.]
>>>> [node03.cluster:15885] MCW rank 5 bound to socket 0[core 4[hwt
>>>> 0]],
>>>> socket
>>>> 0[core 5[hwt 0]], socket 0[core 6[hwt 0]], so
>>>> cket 0[core 7[hwt 0]]:
>>>>
>>>>
>>>> [././././B/B/B/B][./././././././.][./././././././.][./././././././.]
>>>> [node03.cluster:15885] MCW rank 6 bound to socket 0[core 4[hwt
>>>> 0]],
>>>> socket
>>>> 0[core 5[hwt 0]], socket 0[core 6[hwt 0]], so
>>>> cket 0[core 7[hwt 0]]:
>>>>
>>>>
>>>> [././././B/B/B/B][./././././././.][./././././././.][./././././././.]
>>>> [node03.cluster:15885] MCW rank 7 bound to socket 0[core 4[hwt
>>>> 0]],
>>>> socket
>>>> 0[core 5[hwt 0]], socket 0[core 6[hwt 0]], so
>>>> cket 0[core 7[hwt 0]]:
>>>>
>>>>
>>>> [././././B/B/B/B][./././././././.][./././././././.][./././././././.]
>>>> [node03.cluster:15885] MCW rank 0 bound to socket 0[core 0[hwt
>>>> 0]],
>>>> socket
>>>> 0[core 1[hwt 0]], socket 0[core 2[hwt 0]], so
>>>> cket 0[core 3[hwt 0]]:
>>>>
>>>>
>>>> [B/B/B/B/./././.][./././././././.][./././././././.][./././././././.]
>>>> [node03.cluster:15885] MCW rank 1 bound to socket 0[core 0[hwt
>>>> 0]],
>>>> socket
>>>> 0[core 1[hwt 0]], socket 0[core 2[hwt 0]], so
>>>> cket 0[core 3[hwt 0]]:
>>>>
>>>>
>>>> [B/B/B/B/./././.][./././././././.][./././././././.][./././././././.]
>>>> Hello world from process 6 of 8
>>>> Hello world from process 5 of 8
>>>> Hello world from process 0 of 8
>>>> Hello world from process 7 of 8
>>>> Hello world from process 3 of 8
>>>> Hello world from process 4 of 8
>>>> Hello world from process 2 of 8
>>>> Hello world from process 1 of 8
>>>>
>>>> Regards,
>>>> Tetsuya Mishima
>>>>
>>>> [node03.cluster:15508] Type: Machine Number of child objects:
>>>> 4
>>>> Name=NULL
>>>> total=132358820KB
>>>> Backend=Linux
>>>> OSName=Linux
>>>> OSRelease=2.6.18-308.16.1.el5
>>>> OSVersion="#1 SMP Tue Oct 2 22:01:43 EDT 2012"
>>>> Architecture=x86_64
>>>> Cpuset: 0xffffffff
>>>> Online: 0xffffffff
>>>> Allowed: 0xffffffff
>>>> Bind CPU proc: TRUE
>>>> Bind CPU thread: TRUE
>>>> Bind MEM proc: FALSE
>>>> Bind MEM thread: TRUE
>>>> Type: Socket Number of child objects: 2
>>>> Name=NULL
>>>> total=33071780KB
>>>> CPUModel="AMD Opteron(tm) Processor 6136"
>>>> Cpuset: 0x000000ff
>>>> Online: 0x000000ff
>>>> Allowed: 0x000000ff
>>>> Type: NUMANode Number of child objects: 1
>>>>
>>>>
>>>> [node05.cluster:21750] Type: Machine Number of child objects:
>>>> 2
>>>> Name=NULL
>>>> total=33080072KB
>>>> Backend=Linux
>>>> OSName=Linux
>>>> OSRelease=2.6.18-308.16.1.el5
>>>> OSVersion="#1 SMP Tue Oct 2 22:01:43 EDT 2012"
>>>> Architecture=x86_64
>>>> Cpuset: 0x000000ff
>>>> Online: 0x000000ff
>>>> Allowed: 0x000000ff
>>>> Bind CPU proc: TRUE
>>>> Bind CPU thread: TRUE
>>>> Bind MEM proc: FALSE
>>>> Bind MEM thread: TRUE
>>>> Type: NUMANode Number of child objects: 1
>>>> Name=NULL
>>>> local=16532232KB
>>>> total=16532232KB
>>>> Cpuset: 0x0000000f
>>>> Online: 0x0000000f
>>>> Allowed: 0x0000000f
>>>> Type: Socket Number of child objects: 1
>>>>
>>>>
>>>> Hmm...try adding "-mca rmaps_base_verbose 10 -mca
>>>> ess_base_verbose
>>>> 5"
>>>> to
>>>> your cmd line and let's see what it thinks it found.
>>>>
>>>> On Dec 18, 2013, at 6:55 PM, tmishima_at_[hidden]
>>>> wrote:
>>>>
>>>>
>>>> Hi, I report one more problem with openmpi-1.7.4rc1,
>>>> which is more serious.
>>>>
>>>> For our 32 core nodes(AMD magny cours based) which has
>>>> 8 numa-nodes, "-bind-to numa" does not work. Without
>>>> this option, it works. For your infomation, at the
>>>> bottom of this mail, I added the lstopo information
>>>> of the node.
>>>>
>>>> Regards,
>>>> Tetsuya Mishima
>>>>
>>>> [mishima_at_manage ~]$ qsub -I -l nodes=1:ppn=32>> qsub: waiting for job
> 8352.manage.cluster to start
>>>> qsub: job 8352.manage.cluster ready
>>>>
>>>> [mishima_at_node03 demos]$ mpirun -np 8 -report-bindings
>>>> -bind-to
>>>> numa
>>>> myprog
>>>> [node03.cluster:15316] [[37582,0],0] bind:upward target
>>>> NUMANode
>>>> type
>>>> Machine
>>>>
>>>>
>>>>
>>>>
>>>>
>>>
> --------------------------------------------------------------------------
>>>> A request was made to bind to NUMA, but an appropriate
>>>> target
>>>> could
>>>> not
>>>> be found on node node03.
>>>>
>>>>
>>>>
>>>>
>>>>
>>>
> --------------------------------------------------------------------------
>>>> [mishima_at_node03 ~]$ cd ~/Desktop/openmpi-1.7/demos/
>>>> [mishima_at_node03 demos]$ mpirun -np 8 -report-bindings myprog
>>>> [node03.cluster:15282] MCW rank 2 bound to socket 1[core 8
>>>> [hwt
>>>> 0]]:
>>>> [./././././././.][B/././././././.][./././././././.][
>>>> ./././././././.]>>>>>>>>>>>> [node03.cluster:15282] MCW rank
>>>> 3 bound to socket 1[core 9[hwt
>>>> 0]]:
>>>> [./././././././.][./B/./././././.][./././././././.][
>>>> ./././././././.]
>>>> [node03.cluster:15282] MCW rank 4 bound to socket 2[core 16
>>>> [hwt
>>>> 0]]:
>>>> [./././././././.][./././././././.][B/././././././.]
>>>> [./././././././.]
>>>> [node03.cluster:15282] MCW rank 5 bound to socket 2[core 17
>>>> [hwt
>>>> 0]]:
>>>> [./././././././.][./././././././.][./B/./././././.]
>>>> [./././././././.]
>>>> [node03.cluster:15282] MCW rank 6 bound to socket 3[core 24
>>>> [hwt
>>>> 0]]:
>>>> [./././././././.][./././././././.][./././././././.]
>>>> [B/././././././.]
>>>> [node03.cluster:15282] MCW rank 7 bound to socket 3[core 25
>>>> [hwt
>>>> 0]]:
>>>> [./././././././.][./././././././.][./././././././.]
>>>> [./B/./././././.]
>>>> [node03.cluster:15282] MCW rank 0 bound to socket 0[core 0
>>>> [hwt
>>>> 0]]:
>>>> [B/././././././.][./././././././.][./././././././.][
>>>> ./././././././.]
>>>> [node03.cluster:15282] MCW rank 1 bound to socket 0[core 1
>>>> [hwt
>>>> 0]]:
>>>> [./B/./././././.][./././././././.][./././././././.][
>>>> ./././././././.]
>>>> Hello world from process 2 of 8
>>>> Hello world from process 5 of 8
>>>> Hello world from process 4 of 8
>>>> Hello world from process 3 of 8>>>>>>>>>> Hello world from
>>>> process 1 of 8
>>>> Hello world from process 7 of 8
>>>> Hello world from process 6 of 8
>>>> Hello world from process 0 of 8
>>>> [mishima_at_node03 demos]$ ~/opt/hwloc/bin/lstopo-no-graphics
>>>> Machine (126GB)
>>>> Socket L#0 (32GB)
>>>> NUMANode L#0 (P#0 16GB) + L3 L#0 (5118KB)
>>>> L2 L#0 (512KB) + L1d L#0 (64KB) + L1i L#0 (64KB) + Core L#0
>>>> +
>>>> PU
>>>> L#0
>>>> (P#0)
>>>> L2 L#1 (512KB) + L1d L#1 (64KB) + L1i L#1 (64KB) + Core L#1
>>>> +
>>>> PU
>>>> L#1
>>>> (P#1)
>>>> L2 L#2 (512KB) + L1d L#2 (64KB) + L1i L#2 (64KB) + Core L#2
>>>> +
>>>> PU
>>>> L#2
>>>> (P#2)
>>>> L2 L#3 (512KB) + L1d L#3 (64KB) + L1i L#3 (64KB) + Core L#3
>>>> +
>>>> PU
>>>> L#3
>>>> (P#3)
>>>> NUMANode L#1 (P#1 16GB) + L3 L#1 (5118KB)
>>>> L2 L#4 (512KB) + L1d L#4 (64KB) + L1i L#4 (64KB) + Core L#4
>>>> +
>>>> PU
>>>> L#4
>>>> (P#4)
>>>> L2 L#5 (512KB) + L1d L#5 (64KB) + L1i L#5 (64KB) + Core L#5
>>>> +
>>>> PU
>>>> L#5
>>>> (P#5)
>>>> L2 L#6 (512KB) + L1d L#6 (64KB) + L1i L#6 (64KB) + Core L#6
>>>> +
>>>> PU
>>>> L#6
>>>> (P#6)
>>>> L2 L#7 (512KB) + L1d L#7 (64KB) + L1i L#7 (64KB) + Core L#7
>>>> +
>>>> PU>>>>>> L#7
>>>> (P#7)
>>>> Socket L#1 (32GB)
>>>> NUMANode L#2 (P#6 16GB) + L3 L#2 (5118KB)
>>>> L2 L#8 (512KB) + L1d L#8 (64KB) + L1i L#8 (64KB) + Core L#8
>>>> +
>>>> PU
>>>> L#8
>>>> (P#8)
>>>> L2 L#9 (512KB) + L1d L#9 (64KB) + L1i L#9 (64KB) + Core L#9
>>>> +
>>>> PU
>>>> L#9
>>>> (P#9)
>>>> L2 L#10 (512KB) + L1d L#10 (64KB) + L1i L#10 (64KB) + Core
>>>> L#10
>>>> +
>>>> PU
>>>> L#10 (P#10)
>>>> L2 L#11 (512KB) + L1d L#11 (64KB) + L1i L#11 (64KB) + Core
>>>> L#11
>>>> +
>>>> PU
>>>> L#11 (P#11)
>>>> NUMANode L#3 (P#7 16GB) + L3 L#3 (5118KB)
>>>> L2 L#12 (512KB) + L1d L#12 (64KB) + L1i L#12 (64KB) + Core
>>>> L#12
>>>> +
>>>> PU
>>>> L#12 (P#12)
>>>> L2 L#13 (512KB) + L1d L#13 (64KB) + L1i L#13 (64KB) + Core
>>>> L#13
>>>> +
>>>> PU
>>>> L#13 (P#13)
>>>> L2 L#14 (512KB) + L1d L#14 (64KB) + L1i L#14 (64KB) + Core
>>>> L#14
>>>> +
>>>> PU
>>>> L#14 (P#14)
>>>> L2 L#15 (512KB) + L1d L#15 (64KB) + L1i L#15 (64KB) + Core
>>>> L#15
>>>> +
>>>> PU
>>>> L#15 (P#15)
>>>> Socket L#2 (32GB)
>>>> NUMANode L#4 (P#4 16GB) + L3 L#4 (5118KB)
>>>> L2 L#16 (512KB) + L1d L#16 (64KB) + L1i L#16 (64KB) + Core
>>>> L#16
>>>> +
>>>> PU
>>>> L#16 (P#16)
>>>> L2 L#17 (512KB) + L1d L#17 (64KB) + L1i L#17 (64KB) + Core
>>>> L#17
>>>> +
>>>> PU
>>>> L#17 (P#17)> >>>>> L2 L#18 (512KB) + L1d L#18 (64KB) +
>>>> L1i
>>>> L#18 (64KB) + Core L#18
>>>> +
>>>> PU
>>>> L#18 (P#18)
>>>> L2 L#19 (512KB) + L1d L#19 (64KB) + L1i L#19 (64KB) + Core
>>>> L#19
>>>> +
>>>> PU
>>>> L#19 (P#19)
>>>> NUMANode L#5 (P#5 16GB) + L3 L#5 (5118KB)
>>>> L2 L#20 (512KB) + L1d L#20 (64KB) + L1i L#20 (64KB) + Core
>>>> L#20
>>>> +
>>>> PU
>>>> L#20 (P#20)
>>>> L2 L#21 (512KB) + L1d L#21 (64KB) + L1i L#21 (64KB) + Core
>>>> L#21
>>>> +
>>>> PU
>>>> L#21 (P#21)
>>>> L2 L#22 (512KB) + L1d L#22 (64KB) + L1i L#22 (64KB) + Core
>>>> L#22
>>>> +
>>>> PU
>>>> L#22 (P#22)
>>>> L2 L#23 (512KB) + L1d L#23 (64KB) + L1i L#23 (64KB) + Core
>>>> L#23
>>>> +
>>>> PU
>>>> L#23 (P#23)
>>>> Socket L#3 (32GB)
>>>> NUMANode L#6 (P#2 16GB) + L3 L#6 (5118KB)
>>>> L2 L#24 (512KB) + L1d L#24 (64KB) + L1i L#24 (64KB) + Core
>>>> L#24
>>>> +
>>>> PU
>>>> L#24 (P#24)>>>>> L2 L#25 (512KB) + L1d L#25 (64KB) + L1i
>>>> L#25
>>>> (64KB) + Core L#25 +
>>>> PU
>>>> L#25 (P#25)
>>>> L2 L#26 (512KB) + L1d L#26 (64KB) + L1i L#26 (64KB) + Core
>>>> L#26
>>>> +
>>>> PU
>>>> L#26 (P#26)
>>>> L2 L#27 (512KB) + L1d L#27 (64KB) + L1i L#27 (64KB) + Core
>>>> L#27
>>>> +
>>>> PU
>>>> L#27 (P#27)
>>>> NUMANode L#7 (P#3 16GB) + L3 L#7 (5118KB)
>>>> L2 L#28 (512KB) + L1d L#28 (64KB) + L1i L#28 (64KB) + Core
>>>> L#28
>>>> +
>>>> PU
>>>> L#28 (P#28)
>>>> L2 L#29 (512KB) + L1d L#29 (64KB) + L1i L#29 (64KB) + Core
>>>> L#29
>>>> +
>>>> PU
>>>> L#29 (P#29)
>>>> L2 L#30 (512KB) + L1d L#30 (64KB) + L1i L#30 (64KB) + Core
>>>> L#30
>>>> +
>>>> PU
>>>> L#30 (P#30)
>>>> L2 L#31 (512KB) + L1d L#31 (64KB) + L1i L#31 (64KB) + Core
>>>> L#31
>>>> +
>>>> PU
>>>> L#31 (P#31)
>>>> HostBridge L#0
>>>> PCIBridge
>>>> PCI 14e4:1639
>>>> Net L#0 "eth0"
>>>> PCI 14e4:1639
>>>> Net L#1 "eth1"
>>>> PCIBridge
>>>> PCI 14e4:1639
>>>> Net L#2 "eth2"
>>>> PCI 14e4:1639
>>>> Net L#3 "eth3"
>>>> PCIBridge
>>>> PCIBridge
>>>> PCIBridge
>>>> PCI 1000:0072
>>>> Block L#4 "sdb"
>>>> Block L#5 "sda"
>>>> PCI 1002:4390
>>>> Block L#6 "sr0"
>>>> PCIBridge
>>>> PCI 102b:0532
>>>> HostBridge L#7
>>>> PCIBridge
>>>> PCI 15b3:6274
>>>> Net L#7 "ib0"
>>>> OpenFabrics L#8 "mthca0"
>>>>
>>>> _______________________________________________
>>>> users mailing list
>>>> users_at_[hidden]
>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>> _______________________________________________
>>>> users mailing list
>>>> users_at_[hidden]>>
>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>> _______________________________________________
>>>> users mailing list
>>>> users_at_[hidden]
>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>> _______________________________________________
>>>> users mailing list
>>>> users_at_[hidden]
>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>> _______________________________________________
>>>> users mailing list
>>>> users_at_[hidden]
>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>> _______________________________________________
>>>> users mailing list
>>>> users_at_[hidden]
>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>> _______________________________________________
>>>> users mailing list
>>>> users_at_[hidden]
>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>> _______________________________________________
>>>> users mailing list
>>>> users_at_[hidden]
>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>
>>>> _______________________________________________
>>>> users mailing list
>>>> users_at_[hidden]
>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>
>>>> _______________________________________________
>>>> users mailing list
>>>> users_at_[hidden]
>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>
>>>> _______________________________________________
>>>> users mailing list
>>>> users_at_[hidden]
>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>
>>>> _______________________________________________
>>>> users mailing list
>>>> users_at_[hidden]
>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>
>>>> _______________________________________________
>>>> users mailing list
>>>> users_at_[hidden]
>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>
>>>> _______________________________________________
>>>> users mailing list
>>>> users_at_[hidden]
>>>>
>>>
> http://www.open-mpi.org/mailman/listinfo.cgi/users_______________________________________________
>
>>>
>>>> users mailing list
>>>> users_at_[hidden]http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>
>>> _______________________________________________
>>> users mailing list
>>> users_at_[hidden]
>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>
>> _______________________________________________
>> users mailing list
>> users_at_[hidden]
>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>
> _______________________________________________
> users mailing list
> users_at_[hidden]
> http://www.open-mpi.org/mailman/listinfo.cgi/users