Open MPI logo

Hardware Locality Users' Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Hardware Locality Users mailing list

Subject: Re: [hwloc-users] hwloc-ps output - how to verify process binding on the core level?
From: Brice Goglin (Brice.Goglin_at_[hidden])
Date: 2011-02-13 04:07:33


Le 13/02/2011 04:54, Siew Yin Chan a écrit :
> Good day,
>
> I'm studying the impact of MPI process binding on communication costs
> in my project, and would like to use hwloc-bind to achieve
> fine-grained mapping control. I install hwloc 1.1.1 on a 2-socket
> 4-core machine (with 2 dual-core dies in each socket), and run
> hwloc-ps to verify the binding:
>
> $ mpirun -V
> mpirun (Open MPI) 1.5.1
> $ mpirun -np 4 hwloc-bind socket:0.core:0-3 ./test
>
> hwloc-ps shows the following output:
>
> $ hwloc-ps -p
> 1497 Socket:0 ./test
> 1498 Socket:0 ./test
> 1499 Socket:0 ./test
> 1500 Socket:0 ./test
> $ hwloc-ps -l
> 1497 Socket:0 ./test
> 1498 Socket:0 ./test
> 1499 Socket:0 ./test
> 1500 Socket:0 ./test
> $ hwloc-ps -c
> 1497 0x00000055 ./test
> 1498 0x00000055 ./test
> 1499 0x00000055 ./test
> 1500 0x00000055 ./test
>
>
> Questions:
> 1. Does hwloc-bind map the processes *sequentially* on *successive*
> cores of the socket?
>

Hello,

No. Each hwloc-bind command in the mpirun above doesn't know that there
are other hwloc-bind instances on the same machine. All of them bind
their process to all cores in the first socket.

> 2. How could hwloc-ps help verify this binding, i.e.,
>
> process 1497 (rank 0) on socket.0:core.0
> process 1498 (rank 1) on socket.0:core.1
> process 1499 (rank 2) on socket.0:core.2
> process 1500 (rank 3) on socket.0:core.3
>

(let's assume your mpirun command did what you want)

You would get something like this from hwloc-ps:

1497 Core:0 ./test
1498 Core:1 ./test
1499 Core:2 ./test
1500 Core:0 ./test

These core numbers are the logical core number among the entire machine.
hwloc-ps can't easily show hierarchical location such as socket.core
since there are many possible combinations, especially because of caches.

Actually, you might get L1Cache instead of Core above since hwloc-ps
reports the first object that exactly matches the process binding (and
L1 is above but equal to Core in your machine).

If you want to get other output, I suggest you use hwloc-calc to convert
the hwloc-ps output.

> Equivalently, does the binding of `socket:0.core:0-1
> socket:1.core:0-1' with hwloc-ps showing
>
> $ hwloc-ps -l
> 1315 L2Cache:0 L2Cache:2 ./test
> 1316 L2Cache:0 L2Cache:2 ./test
> 1317 L2Cache:0 L2Cache:2 ./test
> 1318 L2Cache:0 L2Cache:2 ./test
>
> indicate the the following? I.e.,
>
> process 1315 (rank 0) on socket.0:core.0
> process 1316 (rank 1) on socket.0:core.1
> process 1317 (rank 2) on socket.1:core.0
> process 1318 (rank 3) on socket.1:core.1
>

No. Again, all processes are bound to 4 different cores, so hwloc-ps
shows the largest objects containing those cores.

In the end, you want a MPI launcher that takes care of the binding
instead of having to manually bind on the command line. It should be the
case with most MPI launchers nowadays. Once this is ok, hwloc-ps will
report this exact core where you bound. And you might need to play with
hwloc-calc to rewrite the hwloc-ps output as you want.

I am thinking of adding an option to hwloc-calc to help rewriting a
random string into socket:X.core:Y or something like that.

Brice