Le 13/02/2011 04:54, Siew Yin Chan a écrit :
Good day,

I'm studying the impact of MPI process binding on communication costs in my project, and would like to use hwloc-bind to achieve fine-grained mapping control. I install hwloc 1.1.1 on a 2-socket 4-core machine (with 2 dual-core dies in each socket), and run hwloc-ps to verify the binding:

$ mpirun -V
mpirun (Open MPI) 1.5.1
$ mpirun -np 4 hwloc-bind socket:0.core:0-3 ./test

hwloc-ps shows the following output:

$ hwloc-ps -p
1497    Socket:0                ./test
1498    Socket:0                ./test
1499    Socket:0                ./test
1500    Socket:0                ./test
$ hwloc-ps -l
1497    Socket:0                ./test
1498    Socket:0                ./test
1499    Socket:0                ./test
1500    Socket:0                ./test
$ hwloc-ps -c
1497    0x00000055              ./test
1498    0x00000055              ./test
1499    0x00000055              ./test
1500    0x00000055              ./test


Questions: 
1. Does hwloc-bind map the processes *sequentially* on *successive* cores of the socket?

Hello,

No. Each hwloc-bind command in the mpirun above doesn't know that there are other hwloc-bind instances on the same machine. All of them bind their process to all cores in the first socket.

2. How could hwloc-ps help verify this binding, i.e.,

process 1497 (rank 0) on socket.0:core.0
process 1498 (rank 1) on socket.0:core.1
process 1499 (rank 2) on socket.0:core.2
process 1500 (rank 3) on socket.0:core.3

(let's assume your mpirun command did what you want)

You would get something like this from hwloc-ps:

1497    Core:0    ./test
1498    Core:1    ./test
1499    Core:2    ./test
1500    Core:0    ./test

These core numbers are the logical core number among the entire machine. hwloc-ps can't easily show hierarchical location such as socket.core since there are many possible combinations, especially because of caches.

Actually, you might get L1Cache instead of Core above since hwloc-ps reports the first object that exactly matches the process binding (and L1 is above but equal to Core in your machine).

If you want to get other output, I suggest you use hwloc-calc to convert the hwloc-ps output.

Equivalently, does the binding of `socket:0.core:0-1 socket:1.core:0-1' with hwloc-ps showing

$ hwloc-ps -l
1315    L2Cache:0 L2Cache:2             ./test
1316    L2Cache:0 L2Cache:2             ./test
1317    L2Cache:0 L2Cache:2             ./test
1318    L2Cache:0 L2Cache:2             ./test

indicate the the following? I.e.,

process 1315 (rank 0) on socket.0:core.0
process 1316 (rank 1) on socket.0:core.1
process 1317 (rank 2) on socket.1:core.0
process 1318 (rank 3) on socket.1:core.1

No. Again, all processes are bound to 4 different cores, so hwloc-ps shows the largest objects containing those cores.


In the end, you want a MPI launcher that takes care of the binding instead of having to manually bind on the command line. It should be the case with most MPI launchers nowadays. Once this is ok, hwloc-ps will report this exact core where you bound. And you might need to play with hwloc-calc to rewrite the hwloc-ps output as you want.

I am thinking of adding an option to hwloc-calc to help rewriting a random string into socket:X.core:Y or something like that.

Brice