Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: Re: [OMPI users] CPU binding
From: Panos Labropoulos (panos.labropoulos_at_[hidden])
Date: 2013-10-02 20:54:23


Dear Jeff,

Thanks for the reply.

I will forward my question there.

Best Regards,
  Panos Labropoulos

On Thu, Oct 3, 2013 at 2:14 AM, Jeff Squyres (jsquyres)
<jsquyres_at_[hidden]>wrote:

> This seems to be a question about hwloc, not about Open MPI.
>
> To clarify, hwloc is a sub-project of Open MPI, but it has its own mailing
> list. Would you mind re-directing your question over there?
>
> http://www.open-mpi.org/community/lists/hwloc.php
>
> Thanks!
>
>
>
> On Oct 2, 2013, at 7:32 PM, Panos Labropoulos <
> panos.labropoulos_at_[hidden]> wrote:
>
> > Hallo,
> >
> > We seem to be unable to to set the cpu binding on a cluster consisting
> of Dell M420/M610 systems:
> >
> > [jallan_at_hpc21 ~]$ cat report-bindings.sh #!/bin/sh
> >
> > bitmap=`hwloc-bind --get -p`
> > friendly=`hwloc-calc -p -H socket.core.pu $bitmap`
> >
> > echo "MCW rank $OMPI_COMM_WORLD_RANK (`hostname`): $friendly"
> > exit 0
> >
> >
> > [jallan_at_hpc27 ~]$ hwloc-bind -v socket:0.core:0 -l ./report-bindings.sh
> > using object #0 depth 2 below cpuset 0x000000ff
> > using object #0 depth 6 below cpuset 0x00000080
> > adding 0x00000080 to 0x0
> > adding 0x00000080 to 0x0
> > assuming the command starts at ./report-bindings.sh
> > binding on cpu set 0x00000080
> > MCW rank (hpc27): Socket:0.Core:10.PU:7
> > [jallan_at_hpc27 ~]$ hwloc-bind -v socket:1.core:0 -l ./report-bindings.sh
> > object #1 depth 2 (type socket) below cpuset 0x000000ff does not exist
> > adding 0x0 to 0x0
> > assuming the command starts at ./report-bindings.sh
> > MCW rank (hpc27): Socket:0.Core:10.PU:7
> >
> >
> > The topology of this system looks a bit strange:
> >
> > [jallan_at_hpc21 ~]$ lstopo --no-io
> > Machine (24GB)
> > NUMANode L#0 (P#0 24GB)
> > NUMANode L#1 (P#1) + Socket L#0 + L3 L#0 (15MB) + L2 L#0 (256KB) + L1
> > L#0 (32KB) + Core L#0 + PU L#0 (P#11)
> > [jallan_at_hpc21 ~]$
> >
> >
> > Using Open MPI 1.4.4:
> >
> > http://pastebin.com/VsZS2q3R
> >
> > For some reason the binding cannot be set. We also tried Open MPI 1.6.5
> and 1.7.3 with similar results.
> >
> > This is the output from a local SMP system:
> >
> > [panos_at_demo ~]$ hwloc-bind -v socket:1.core:0 -l ./report-bindings.sh
> using object #1 depth 2 below cpuset 0x00000003 using object #0 depth 6
> below cpuset 0x00000002 adding 0x00000002 to 0x0 adding 0x00000002 to 0x0
> assuming the command starts at ./report-bindings.sh binding on cpu set
> 0x00000002 MCW rank (demo): Socket:1.Core:0.PU:1 [panos_at_demo ~]$
> hwloc-bind -v socket:0.core:0 -l ./report-bindings.sh using object #0
> depth 2 below cpuset 0x00000003 using object #0 depth 6 below cpuset
> 0x00000001 adding 0x00000001 to 0x0 adding 0x00000001 to 0x0 assuming the
> command starts at ./report-bindings.sh binding on cpu set 0x00000001 MCW
> rank (demo): Socket:0.Core:0.PU:0
> >
> >
> > The MPI binding output is formatted a bit different as this nodes runs
> Open MPI 1.6.5:
> >
> > [panos_at_demo ~]$ `which mpiexec` --report-bindings --bind-to-core
> > --bycore -mca btl ^openib -np 4 -hostfile ./hplnodes2 -x
> > LD_LIBRARY_PATH -x PATH /cm/shared/apps/hpl/2.1/xhpl
> > [demo:25615] MCW rank 0 bound to socket 0[core 0]: [B][.] [demo:25615]
> MCW rank 2 bound to socket 1[core 0]: [.][B] [node003:08454] MCW rank 1
> bound to socket 0[core 0]: [B .] [node003:08454] MCW rank 3 bound to socket
> 0[core 1]: [. B] [panos_at_demo ~]$ module load hwloc
> >
> >
> >
> > [panos_at_demo ~]$ lstopo -l
> > Machine (4095MB)
> > NUMANode L#0 (P#0 2048MB) + Socket L#0 + L2 L#0 (1024KB) + L1d L#0
> > (64KB) + L1i L#0 (64KB) + Core L#0 + PU L#0 (P#0)
> > NUMANode L#1 (P#1 2048MB) + Socket L#1 + L2 L#1 (1024KB) + L1d L#1
> > (64KB) + L1i L#1 (64KB) + Core L#1 + PU L#1 (P#1)
> >
> > Any help will be appreciated.
> >
> > Kind Regards,
> > Panos Labropoulos
> > _______________________________________________
> > users mailing list
> > users_at_[hidden]
> > http://www.open-mpi.org/mailman/listinfo.cgi/users
>
>
> --
> Jeff Squyres
> jsquyres_at_[hidden]
> For corporate legal information go to:
> http://www.cisco.com/web/about/doing_business/legal/cri/
>
> _______________________________________________
> users mailing list
> users_at_[hidden]
> http://www.open-mpi.org/mailman/listinfo.cgi/users
>