Le 13/02/2011 04:54, Siew Yin Chan a écrit :
| Good
day,
I'm
studying the impact of MPI process binding on communication costs in my
project, and would like to use hwloc-bind to achieve fine-grained
mapping control. I install hwloc 1.1.1 on a 2-socket 4-core machine
(with 2 dual-core dies in each socket), and run hwloc-ps to verify the
binding:
$ mpirun -V
mpirun (Open MPI) 1.5.1
$
mpirun -np 4 hwloc-bind socket:0.core:0-3 ./test
hwloc-ps
shows the following output:
$ hwloc-ps -p
1497 Socket:0 ./test
1498 Socket:0 ./test
1499 Socket:0 ./test
1500 Socket:0 ./test
$ hwloc-ps -l
1497 Socket:0 ./test
1498 Socket:0 ./test
1499 Socket:0 ./test
1500 Socket:0 ./test
$ hwloc-ps -c
1497 0x00000055 ./test
1498 0x00000055 ./test
1499 0x00000055 ./test
1500 0x00000055 ./test
Questions:
1. Does hwloc-bind map
the processes *sequentially* on *successive* cores of the socket?
|
Hello,
No. Each hwloc-bind command in the mpirun above doesn't know that there
are other hwloc-bind instances on the same machine. All of them bind
their process to all cores in the first socket.
=> Agree. For socket:0.core:0-3 , hwloc will bind the MPI processes to all cores in the first socket. But how are the individual processes mapped on these cores? Will it be in this order:
rank 0 à core 0
rank 1 à core 1
rank 2 à core 2
rank 3 à core 3
Or in this *arbitrary* order:
rank 0 à core
1
rank 1 à core 3
rank 2 à core 0
rank 3 à core 2
|
2. How could hwloc-ps
help verify this binding, i.e.,
process 1497 (rank 0) on
socket.0:core.0
process 1498 (rank
1) on socket.0:core.1
process 1499 (rank
2) on socket.0:core.2
process
1500 (rank
3) on socket.0:core.3
|
(let's assume your mpirun command did what you want)
You would get something like this from hwloc-ps:
1497 Core:0 ./test
These core numbers are the logical core number among the entire
machine. hwloc-ps can't easily show hierarchical location such as
socket.core since there are many possible combinations, especially
because of caches.
Actually, you might get L1Cache instead of Core above since hwloc-ps
reports the first object that exactly matches the process binding (and
L1 is above but equal to Core in your machine).
If you want to get other output, I suggest you use hwloc-calc to
convert the hwloc-ps output.
|
Equivalently,
does the binding of `socket:0.core:0-1 socket:1.core:0-1' with hwloc-ps
showing
$
hwloc-ps -l
1315
L2Cache:0 L2Cache:2 ./test
1316
L2Cache:0 L2Cache:2 ./test
1317
L2Cache:0 L2Cache:2 ./test
1318
L2Cache:0 L2Cache:2 ./test
indicate the
the following? I.e.,
process 1315 (rank
0) on socket.0:core.0
process 1316 (rank
1) on socket.0:core.1
process 1317 (rank
2) on socket.1:core.0
process 1318 (rank
3) on socket.1:core.1
|
No. Again, all processes are bound to 4 different cores, so hwloc-ps
shows the largest objects containing those cores.
In the end, you want a MPI launcher that takes care of the binding
instead of having to manually bind on the command line. It should be
the case with most MPI launchers nowadays. Once this is ok, hwloc-ps
will report this exact core where you bound. And you might need to play
with hwloc-calc to rewrite the hwloc-ps output as you want.
I am thinking of adding an option to hwloc-calc to help rewriting a
random string into socket:X.core:Y or something like that.
Brice