Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: Re: [OMPI users] "-bind-to numa" of openmpi-1.7.4rc1 dosen't work for our magny cours based 32 core node
From: tmishima_at_[hidden]
Date: 2013-12-20 05:33:09


Hi Ralph,

The numa-node in AMD Mangy-Cours/Interlagos is so called cc(cache
coherent)NUMA,
which seems to be a little bit different from the traditional numa defined
in openmpi.

I notice that ccNUMA object is almost same as L3cache object.
So "-bind-to l3cache" or "-map-by l3cache" is valid for what I want to do.
Therefore, "do not touch it" is one of the solution, I think ...

Anyway, mixing up these two types of numa is the problem.

Regards,
Tetsuya Mishima

> I can wait it'll be fixed in 1.7.5 or later, because putting "-bind-to
> numa"
> and "-map-by numa" at the same time works as a workaround.
>
> Thanks,
> Tetsuya Mishima
>
> > Yeah, it will impact everything that uses hwloc topology maps, I fear.
> >
> > One side note: you'll need to add --hetero-nodes to your cmd line. If
we
> don't see that, we assume that all the node topologies are identical -
> which clearly isn't true here.
> >
> > I'll try to resolve the hier inversion over the holiday - won't be for
> 1.7.4, but hopefully for 1.7.5
> >
> > Thanks
> > Ralph
> >
> > On Dec 18, 2013, at 9:44 PM, tmishima_at_[hidden] wrote:
> >
> > >
> > >
> > > I think it's normal for AMD opteron having 8/16 cores such as
> > > magny cours or interlagos. Because it usually has 2 numa nodes
> > > in a cpu(socket), numa-node can not include a socket. This type
> > > of hierarchy would be natural.
> > >
> > > (node03 is Dell PowerEdge R815 and maybe quite common, I guess)
> > >
> > > By the way, I think this inversion should affect rmaps_lama mapping.
> > >
> > > Tetsuya Mishima
> > >
> > >> Ick - yeah, that would be a problem. I haven't seen that type of
> > > hierarchical inversion before - is node03 a different type of chip?
> > >>
> > >> Might take awhile for me to adjust the code to handle hier
> > > inversion... :-(
> > >>
> > >> On Dec 18, 2013, at 9:05 PM, tmishima_at_[hidden] wrote:
> > >>
> > >>>
> > >>>
> > >>> Hi Ralph,
> > >>>
> > >>> I found the reason. I attached the main part of output with 32
> > >>> core node(node03) and 8 core node(node05) at the bottom.
> > >>>
> > >>> From this information, socket of node03 includes numa-node.
> > >>> On the other hand, numa-node of node05 includes socket.
> > >>> The direction of object tree is opposite.
> > >>>
> > >>> Since "-map-by socket" may be assumed as default,
> > >>> for node05, "-bind-to numa and -map-by socket" means
> > >>> upward search. For node03, this should be downward.
> > >>>
> > >>> I guess that openmpi-1.7.4rc1 will always assume numa-node
> > >>> includes socket. Is it right? Then, upward search is assumed
> > >>> in orte_rmaps_base_compute_bindings even for node03 when I
> > >>> put "-bind-to numa and -map-by socket" option.
> > >>>
> > >>> [node03.cluster:15508] [[38286,0],0] rmaps:base:compute_usage
> > >>> [node03.cluster:15508] mca:rmaps: compute bindings for job
[38286,1]
> > > with
> > >>> policy NUMA
> > >>> [node03.cluster:15508] mca:rmaps: bind upwards for job [38286,1]
with
> > >>> bindings NUMA
> > >>> [node03.cluster:15508] [[38286,0],0] bind:upward target NUMANode
type
> > >>> Machine
> > >>>
> > >>> That's the reason of this trouble. Therefore, adding "-map-by core"
> > > works.
> > >>> (mapping pattern seems to be strange ...)
> > >>>
> > >>> [mishima_at_node03 demos]$ mpirun -np 8 -bind-to numa -map-by core
> > >>> -report-bindings myprog
> > >>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode
type
> > > Cache
> > >>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode
type
> > > Cache
> > >>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode
type
> > > Cache
> > >>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode
type
> > >>> NUMANode
> > >>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode
type
> > > Cache
> > >>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode
type
> > > Cache
> > >>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode
type
> > > Cache
> > >>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode
type
> > >>> NUMANode
> > >>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode
type
> > > Cache
> > >>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode
type
> > > Cache
> > >>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode
type
> > > Cache
> > >>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode
type
> > >>> NUMANode
> > >>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode
type
> > > Cache
> > >>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode
type
> > > Cache
> > >>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode
type
> > > Cache
> > >>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode
type
> > >>> NUMANode
> > >>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode
type
> > > Cache
> > >>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode
type
> > > Cache
> > >>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode
type
> > > Cache
> > >>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode
type
> > >>> NUMANode
> > >>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode
type
> > > Cache
> > >>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode
type
> > > Cache
> > >>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode
type
> > > Cache
> > >>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode
type
> > >>> NUMANode
> > >>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode
type
> > > Cache
> > >>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode
type
> > > Cache
> > >>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode
type
> > > Cache
> > >>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode
type
> > >>> NUMANode
> > >>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode
type
> > > Cache
> > >>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode
type
> > > Cache
> > >>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode
type
> > > Cache
> > >>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode
type
> > >>> NUMANode
> > >>> [node03.cluster:15885] MCW rank 2 bound to socket 0[core 0[hwt 0]],
> > > socket
> > >>> 0[core 1[hwt 0]], socket 0[core 2[hwt 0]], so
> > >>> cket 0[core 3[hwt 0]]:
> > >>>
[B/B/B/B/./././.][./././././././.][./././././././.][./././././././.]
> > >>> [node03.cluster:15885] MCW rank 3 bound to socket 0[core 0[hwt 0]],
> > > socket
> > >>> 0[core 1[hwt 0]], socket 0[core 2[hwt 0]], so
> > >>> cket 0[core 3[hwt 0]]:
> > >>>
[B/B/B/B/./././.][./././././././.][./././././././.][./././././././.]
> > >>> [node03.cluster:15885] MCW rank 4 bound to socket 0[core 4[hwt 0]],
> > > socket
> > >>> 0[core 5[hwt 0]], socket 0[core 6[hwt 0]], so
> > >>> cket 0[core 7[hwt 0]]:
> > >>>
[././././B/B/B/B][./././././././.][./././././././.][./././././././.]
> > >>> [node03.cluster:15885] MCW rank 5 bound to socket 0[core 4[hwt 0]],
> > > socket
> > >>> 0[core 5[hwt 0]], socket 0[core 6[hwt 0]], so
> > >>> cket 0[core 7[hwt 0]]:
> > >>>
[././././B/B/B/B][./././././././.][./././././././.][./././././././.]
> > >>> [node03.cluster:15885] MCW rank 6 bound to socket 0[core 4[hwt 0]],
> > > socket
> > >>> 0[core 5[hwt 0]], socket 0[core 6[hwt 0]], so
> > >>> cket 0[core 7[hwt 0]]:
> > >>>
[././././B/B/B/B][./././././././.][./././././././.][./././././././.]
> > >>> [node03.cluster:15885] MCW rank 7 bound to socket 0[core 4[hwt 0]],
> > > socket
> > >>> 0[core 5[hwt 0]], socket 0[core 6[hwt 0]], so
> > >>> cket 0[core 7[hwt 0]]:
> > >>>
[././././B/B/B/B][./././././././.][./././././././.][./././././././.]
> > >>> [node03.cluster:15885] MCW rank 0 bound to socket 0[core 0[hwt 0]],
> > > socket
> > >>> 0[core 1[hwt 0]], socket 0[core 2[hwt 0]], so
> > >>> cket 0[core 3[hwt 0]]:
> > >>>
[B/B/B/B/./././.][./././././././.][./././././././.][./././././././.]
> > >>> [node03.cluster:15885] MCW rank 1 bound to socket 0[core 0[hwt 0]],
> > > socket
> > >>> 0[core 1[hwt 0]], socket 0[core 2[hwt 0]], so
> > >>> cket 0[core 3[hwt 0]]:
> > >>>
[B/B/B/B/./././.][./././././././.][./././././././.][./././././././.]
> > >>> Hello world from process 6 of 8
> > >>> Hello world from process 5 of 8
> > >>> Hello world from process 0 of 8
> > >>> Hello world from process 7 of 8
> > >>> Hello world from process 3 of 8
> > >>> Hello world from process 4 of 8
> > >>> Hello world from process 2 of 8
> > >>> Hello world from process 1 of 8
> > >>>
> > >>> Regards,
> > >>> Tetsuya Mishima
> > >>>
> > >>> [node03.cluster:15508] Type: Machine Number of child objects: 4
> > >>> Name=NULL
> > >>> total=132358820KB
> > >>> Backend=Linux
> > >>> OSName=Linux
> > >>> OSRelease=2.6.18-308.16.1.el5
> > >>> OSVersion="#1 SMP Tue Oct 2 22:01:43 EDT 2012"
> > >>> Architecture=x86_64
> > >>> Cpuset: 0xffffffff
> > >>> Online: 0xffffffff
> > >>> Allowed: 0xffffffff
> > >>> Bind CPU proc: TRUE
> > >>> Bind CPU thread: TRUE
> > >>> Bind MEM proc: FALSE
> > >>> Bind MEM thread: TRUE
> > >>> Type: Socket Number of child objects: 2
> > >>> Name=NULL
> > >>> total=33071780KB
> > >>> CPUModel="AMD Opteron(tm) Processor 6136"
> > >>> Cpuset: 0x000000ff
> > >>> Online: 0x000000ff
> > >>> Allowed: 0x000000ff
> > >>> Type: NUMANode Number of child objects: 1
> > >>>
> > >>>
> > >>> [node05.cluster:21750] Type: Machine Number of child objects: 2
> > >>> Name=NULL
> > >>> total=33080072KB
> > >>> Backend=Linux
> > >>> OSName=Linux
> > >>> OSRelease=2.6.18-308.16.1.el5
> > >>> OSVersion="#1 SMP Tue Oct 2 22:01:43 EDT 2012"
> > >>> Architecture=x86_64
> > >>> Cpuset: 0x000000ff
> > >>> Online: 0x000000ff
> > >>> Allowed: 0x000000ff
> > >>> Bind CPU proc: TRUE
> > >>> Bind CPU thread: TRUE
> > >>> Bind MEM proc: FALSE
> > >>> Bind MEM thread: TRUE
> > >>> Type: NUMANode Number of child objects: 1
> > >>> Name=NULL
> > >>> local=16532232KB
> > >>> total=16532232KB
> > >>> Cpuset: 0x0000000f
> > >>> Online: 0x0000000f
> > >>> Allowed: 0x0000000f
> > >>> Type: Socket Number of child objects: 1
> > >>>
> > >>>
> > >>>> Hmm...try adding "-mca rmaps_base_verbose 10 -mca ess_base_verbose
> 5"
> > > to
> > >>> your cmd line and let's see what it thinks it found.
> > >>>>
> > >>>>
> > >>>> On Dec 18, 2013, at 6:55 PM, tmishima_at_[hidden] wrote:
> > >>>>
> > >>>>>
> > >>>>>
> > >>>>> Hi, I report one more problem with openmpi-1.7.4rc1,
> > >>>>> which is more serious.
> > >>>>>
> > >>>>> For our 32 core nodes(AMD magny cours based) which has
> > >>>>> 8 numa-nodes, "-bind-to numa" does not work. Without
> > >>>>> this option, it works. For your infomation, at the
> > >>>>> bottom of this mail, I added the lstopo information
> > >>>>> of the node.
> > >>>>>
> > >>>>> Regards,
> > >>>>> Tetsuya Mishima
> > >>>>>
> > >>>>> [mishima_at_manage ~]$ qsub -I -l nodes=1:ppn=32
> > >>>>> qsub: waiting for job 8352.manage.cluster to start
> > >>>>> qsub: job 8352.manage.cluster ready
> > >>>>>
> > >>>>> [mishima_at_node03 demos]$ mpirun -np 8 -report-bindings -bind-to
numa
> > >>> myprog
> > >>>>> [node03.cluster:15316] [[37582,0],0] bind:upward target NUMANode
> type
> > >>>>> Machine
> > >>>>>
> > >>>
> > >
>
--------------------------------------------------------------------------
> > >>>>> A request was made to bind to NUMA, but an appropriate target
could
> > > not
> > >>>>> be found on node node03.
> > >>>>>
> > >>>
> > >
>
--------------------------------------------------------------------------
> > >>>>> [mishima_at_node03 ~]$ cd ~/Desktop/openmpi-1.7/demos/
> > >>>>> [mishima_at_node03 demos]$ mpirun -np 8 -report-bindings myprog
> > >>>>> [node03.cluster:15282] MCW rank 2 bound to socket 1[core 8[hwt
0]]:
> > >>>>> [./././././././.][B/././././././.][./././././././.][
> > >>>>> ./././././././.]
> > >>>>> [node03.cluster:15282] MCW rank 3 bound to socket 1[core 9[hwt
0]]:
> > >>>>> [./././././././.][./B/./././././.][./././././././.][
> > >>>>> ./././././././.]
> > >>>>> [node03.cluster:15282] MCW rank 4 bound to socket 2[core 16[hwt
> 0]]:
> > >>>>> [./././././././.][./././././././.][B/././././././.]
> > >>>>> [./././././././.]
> > >>>>> [node03.cluster:15282] MCW rank 5 bound to socket 2[core 17[hwt
> 0]]:
> > >>>>> [./././././././.][./././././././.][./B/./././././.]
> > >>>>> [./././././././.]
> > >>>>> [node03.cluster:15282] MCW rank 6 bound to socket 3[core 24[hwt
> 0]]:
> > >>>>> [./././././././.][./././././././.][./././././././.]
> > >>>>> [B/././././././.]
> > >>>>> [node03.cluster:15282] MCW rank 7 bound to socket 3[core 25[hwt
> 0]]:
> > >>>>> [./././././././.][./././././././.][./././././././.]
> > >>>>> [./B/./././././.]
> > >>>>> [node03.cluster:15282] MCW rank 0 bound to socket 0[core 0[hwt
0]]:
> > >>>>> [B/././././././.][./././././././.][./././././././.][
> > >>>>> ./././././././.]
> > >>>>> [node03.cluster:15282] MCW rank 1 bound to socket 0[core 1[hwt
0]]:
> > >>>>> [./B/./././././.][./././././././.][./././././././.][
> > >>>>> ./././././././.]
> > >>>>> Hello world from process 2 of 8
> > >>>>> Hello world from process 5 of 8
> > >>>>> Hello world from process 4 of 8
> > >>>>> Hello world from process 3 of 8
> > >>>>> Hello world from process 1 of 8
> > >>>>> Hello world from process 7 of 8
> > >>>>> Hello world from process 6 of 8
> > >>>>> Hello world from process 0 of 8
> > >>>>> [mishima_at_node03 demos]$ ~/opt/hwloc/bin/lstopo-no-graphics
> > >>>>> Machine (126GB)
> > >>>>> Socket L#0 (32GB)
> > >>>>> NUMANode L#0 (P#0 16GB) + L3 L#0 (5118KB)
> > >>>>> L2 L#0 (512KB) + L1d L#0 (64KB) + L1i L#0 (64KB) + Core L#0 +
PU
> > >>> L#0
> > >>>>> (P#0)
> > >>>>> L2 L#1 (512KB) + L1d L#1 (64KB) + L1i L#1 (64KB) + Core L#1 +
PU
> > >>> L#1
> > >>>>> (P#1)
> > >>>>> L2 L#2 (512KB) + L1d L#2 (64KB) + L1i L#2 (64KB) + Core L#2 +
PU
> > >>> L#2
> > >>>>> (P#2)
> > >>>>> L2 L#3 (512KB) + L1d L#3 (64KB) + L1i L#3 (64KB) + Core L#3 +
PU
> > >>> L#3
> > >>>>> (P#3)
> > >>>>> NUMANode L#1 (P#1 16GB) + L3 L#1 (5118KB)
> > >>>>> L2 L#4 (512KB) + L1d L#4 (64KB) + L1i L#4 (64KB) + Core L#4 +
PU
> > >>> L#4
> > >>>>> (P#4)
> > >>>>> L2 L#5 (512KB) + L1d L#5 (64KB) + L1i L#5 (64KB) + Core L#5 +
PU
> > >>> L#5
> > >>>>> (P#5)
> > >>>>> L2 L#6 (512KB) + L1d L#6 (64KB) + L1i L#6 (64KB) + Core L#6 +
PU
> > >>> L#6
> > >>>>> (P#6)
> > >>>>> L2 L#7 (512KB) + L1d L#7 (64KB) + L1i L#7 (64KB) + Core L#7 +
PU
> > >>> L#7
> > >>>>> (P#7)
> > >>>>> Socket L#1 (32GB)
> > >>>>> NUMANode L#2 (P#6 16GB) + L3 L#2 (5118KB)
> > >>>>> L2 L#8 (512KB) + L1d L#8 (64KB) + L1i L#8 (64KB) + Core L#8 +
PU
> > >>> L#8
> > >>>>> (P#8)
> > >>>>> L2 L#9 (512KB) + L1d L#9 (64KB) + L1i L#9 (64KB) + Core L#9 +
PU
> > >>> L#9
> > >>>>> (P#9)
> > >>>>> L2 L#10 (512KB) + L1d L#10 (64KB) + L1i L#10 (64KB) + Core
L#10
> +
> > >>> PU
> > >>>>> L#10 (P#10)
> > >>>>> L2 L#11 (512KB) + L1d L#11 (64KB) + L1i L#11 (64KB) + Core
L#11
> +
> > >>> PU
> > >>>>> L#11 (P#11)
> > >>>>> NUMANode L#3 (P#7 16GB) + L3 L#3 (5118KB)
> > >>>>> L2 L#12 (512KB) + L1d L#12 (64KB) + L1i L#12 (64KB) + Core
L#12
> +
> > >>> PU
> > >>>>> L#12 (P#12)
> > >>>>> L2 L#13 (512KB) + L1d L#13 (64KB) + L1i L#13 (64KB) + Core
L#13
> +
> > >>> PU
> > >>>>> L#13 (P#13)
> > >>>>> L2 L#14 (512KB) + L1d L#14 (64KB) + L1i L#14 (64KB) + Core
L#14
> +
> > >>> PU
> > >>>>> L#14 (P#14)
> > >>>>> L2 L#15 (512KB) + L1d L#15 (64KB) + L1i L#15 (64KB) + Core
L#15
> +
> > >>> PU
> > >>>>> L#15 (P#15)
> > >>>>> Socket L#2 (32GB)
> > >>>>> NUMANode L#4 (P#4 16GB) + L3 L#4 (5118KB)
> > >>>>> L2 L#16 (512KB) + L1d L#16 (64KB) + L1i L#16 (64KB) + Core
L#16
> +
> > >>> PU
> > >>>>> L#16 (P#16)
> > >>>>> L2 L#17 (512KB) + L1d L#17 (64KB) + L1i L#17 (64KB) + Core
L#17
> +
> > >>> PU
> > >>>>> L#17 (P#17)> >>>>> L2 L#18 (512KB) + L1d L#18 (64KB) + L1i
L#18 (64KB) + Core L#18
> +
> > >>> PU
> > >>>>> L#18 (P#18)
> > >>>>> L2 L#19 (512KB) + L1d L#19 (64KB) + L1i L#19 (64KB) + Core
L#19
> +
> > >>> PU
> > >>>>> L#19 (P#19)
> > >>>>> NUMANode L#5 (P#5 16GB) + L3 L#5 (5118KB)
> > >>>>> L2 L#20 (512KB) + L1d L#20 (64KB) + L1i L#20 (64KB) + Core
L#20
> +
> > >>> PU
> > >>>>> L#20 (P#20)
> > >>>>> L2 L#21 (512KB) + L1d L#21 (64KB) + L1i L#21 (64KB) + Core
L#21
> +
> > >>> PU
> > >>>>> L#21 (P#21)
> > >>>>> L2 L#22 (512KB) + L1d L#22 (64KB) + L1i L#22 (64KB) + Core
L#22
> +
> > >>> PU
> > >>>>> L#22 (P#22)
> > >>>>> L2 L#23 (512KB) + L1d L#23 (64KB) + L1i L#23 (64KB) + Core
L#23
> +
> > >>> PU
> > >>>>> L#23 (P#23)
> > >>>>> Socket L#3 (32GB)
> > >>>>> NUMANode L#6 (P#2 16GB) + L3 L#6 (5118KB)
> > >>>>> L2 L#24 (512KB) + L1d L#24 (64KB) + L1i L#24 (64KB) + Core
L#24
> +
> > >>> PU
> > >>>>> L#24 (P#24)>>>>> L2 L#25 (512KB) + L1d L#25 (64KB) + L1i L#25
> (64KB) + Core L#25 +
> > >>> PU
> > >>>>> L#25 (P#25)
> > >>>>> L2 L#26 (512KB) + L1d L#26 (64KB) + L1i L#26 (64KB) + Core
L#26
> +
> > >>> PU
> > >>>>> L#26 (P#26)
> > >>>>> L2 L#27 (512KB) + L1d L#27 (64KB) + L1i L#27 (64KB) + Core
L#27
> +
> > >>> PU
> > >>>>> L#27 (P#27)
> > >>>>> NUMANode L#7 (P#3 16GB) + L3 L#7 (5118KB)
> > >>>>> L2 L#28 (512KB) + L1d L#28 (64KB) + L1i L#28 (64KB) + Core
L#28
> +
> > >>> PU
> > >>>>> L#28 (P#28)
> > >>>>> L2 L#29 (512KB) + L1d L#29 (64KB) + L1i L#29 (64KB) + Core
L#29
> +
> > >>> PU
> > >>>>> L#29 (P#29)
> > >>>>> L2 L#30 (512KB) + L1d L#30 (64KB) + L1i L#30 (64KB) + Core
L#30
> +
> > >>> PU
> > >>>>> L#30 (P#30)
> > >>>>> L2 L#31 (512KB) + L1d L#31 (64KB) + L1i L#31 (64KB) + Core
L#31
> +
> > >>> PU
> > >>>>> L#31 (P#31)
> > >>>>> HostBridge L#0
> > >>>>> PCIBridge
> > >>>>> PCI 14e4:1639
> > >>>>> Net L#0 "eth0"
> > >>>>> PCI 14e4:1639
> > >>>>> Net L#1 "eth1"
> > >>>>> PCIBridge
> > >>>>> PCI 14e4:1639
> > >>>>> Net L#2 "eth2"
> > >>>>> PCI 14e4:1639
> > >>>>> Net L#3 "eth3"
> > >>>>> PCIBridge
> > >>>>> PCIBridge
> > >>>>> PCIBridge
> > >>>>> PCI 1000:0072
> > >>>>> Block L#4 "sdb"
> > >>>>> Block L#5 "sda"
> > >>>>> PCI 1002:4390
> > >>>>> Block L#6 "sr0"
> > >>>>> PCIBridge
> > >>>>> PCI 102b:0532
> > >>>>> HostBridge L#7
> > >>>>> PCIBridge
> > >>>>> PCI 15b3:6274
> > >>>>> Net L#7 "ib0"
> > >>>>> OpenFabrics L#8 "mthca0"
> > >>>>>
> > >>>>> _______________________________________________
> > >>>>> users mailing list
> > >>>>> users_at_[hidden]
> > >>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
> > >>>>
> > >>>> _______________________________________________
> > >>>> users mailing list
> > >>>> users_at_[hidden]>>
> > > http://www.open-mpi.org/mailman/listinfo.cgi/users
> > >>>
> > >>> _______________________________________________
> > >>> users mailing list
> > >>> users_at_[hidden]
> > >>> http://www.open-mpi.org/mailman/listinfo.cgi/users
> > >>
> > >> _______________________________________________
> > >> users mailing list
> > >> users_at_[hidden]
> > >> http://www.open-mpi.org/mailman/listinfo.cgi/users
> > >
> > > _______________________________________________
> > > users mailing list
> > > users_at_[hidden]
> > > http://www.open-mpi.org/mailman/listinfo.cgi/users
> >
> > _______________________________________________
> > users mailing list
> > users_at_[hidden]
> > http://www.open-mpi.org/mailman/listinfo.cgi/users
>
> _______________________________________________
> users mailing list
> users_at_[hidden]
> http://www.open-mpi.org/mailman/listinfo.cgi/users