FWIW: 1.5.5 still doesn't support binding to NUMA regions, for example - and the script doesn't really do anything more than bind to cores. I believe only the trunk provides a more comprehensive set of binding options.

Given the described NUMA layout, I suspect bind-to-NUMA is going to make the biggest difference.

On Mar 30, 2012, at 6:17 AM, Pavel Mezentsev wrote:

You can try running using this script:
#!/bin/bash

s=$(($OMPI_COMM_WORLD_NODE_RANK))

numactl --physcpubind=$((s)) --localalloc ./YOUR_PROG

instead of 'mpirun ... ./YOUR_PROG' run 'mpirun ... ./SCRIPT

I tried this with openmpi-1.5.4 and it helped.

Best regards, Pavel Mezentsev

P.S openmpi-1.5.5 bind processes correctly, so you can try it as well.

2012/3/30 Ralph Castain <rhc@open-mpi.org>
I think you'd have much better luck using the developer's trunk as the binding there is much better - e.g., you can bind to NUMA instead of just cores. The 1.4 binding is pretty limited.

http://www.open-mpi.org/nightly/trunk/

On Mar 30, 2012, at 5:02 AM, Ricardo Fonseca wrote:

> Hi guys
>
> I'm benchmarking our (well tested) parallel code on and AMD based system, featuring 2x AMD Opteron(TM) Processor 6276, with 16 cores each for a total of 32 cores. The system is running Scientific Linux 6.1 and OpenMPI 1.4.5.
>
> When I run a single core job the performance is as expected. However, when I run with 32 processes the performance drops to about 60% (when compared with other systems running the exact same problem, so this is not a code scaling issue). I think this may have to do with core binding / NUMA, but I haven't been able to get any improvement out of the bind-* mpirun options.
>
> Any suggestions?
>
> Thanks in advance,
> Ricardo
>
> P.S: Here's the output of lscpu
>
> Architecture:          x86_64
> CPU op-mode(s):        32-bit, 64-bit
> Byte Order:            Little Endian
> CPU(s):                32
> On-line CPU(s) list:   0-31
> Thread(s) per core:    2
> Core(s) per socket:    8
> CPU socket(s):         2
> NUMA node(s):          4
> Vendor ID:             AuthenticAMD
> CPU family:            21
> Model:                 1
> Stepping:              2
> CPU MHz:               2300.045
> BogoMIPS:              4599.38
> Virtualization:        AMD-V
> L1d cache:             16K
> L1i cache:             64K
> L2 cache:              2048K
> L3 cache:              6144K
> NUMA node0 CPU(s):     0,2,4,6,8,10,12,14
> NUMA node1 CPU(s):     16,18,20,22,24,26,28,30
> NUMA node2 CPU(s):     1,3,5,7,9,11,13,15
> NUMA node3 CPU(s):     17,19,21,23,25,27,29,31
>
> ---
> Ricardo Fonseca
>
> Associate Professor
> GoLP - Grupo de Lasers e Plasmas
> Instituto de Plasmas e Fusão Nuclear
> Instituto Superior Técnico
> Av. Rovisco Pais
> 1049-001 Lisboa
> Portugal
>
> tel: +351 21 8419202
> fax: +351 21 8464455
> web: http://golp.ist.utl.pt/
>
>
> _______________________________________________
> users mailing list
> users@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users


_______________________________________________
users mailing list
users@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users

_______________________________________________
users mailing list
users@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users