On 11/23/2011 10:11 AM, Lukas Razik wrote:

TERRY DONTJE <terry.dontje@oracle.com> wrote:

Can you try running the benchmark with coalescing off?  To do that

    add the following option to your mpirun line "-mca
    btl_openib_use_message_coalescing 0".

I've tried this:

# /usr/mpi/gcc/openmpi-1.4.4/bin/mpirun -np 2
  --mca btl_openib_use_message_coalescing 0
  --mca btl_base_verbose 50
  --mca btl_openib_verbose 1
  -host cluster1,cluster2
    /usr/mpi/gcc/openmpi-1.4.3/tests/osu_benchmarks-3.1.1/osu_latency


And that's the result (which isn't different from the run without "--mca btl_openib_use_message_coalescing 0"):
http://net.razik.de/linux/T5120/openmpi-1.4.4-verbose_no_coalescing.txt

Nuts!!! Ok I am going to have to think about this a little more. Do you have the ability to configure and remake your ompi install? I might want to have you add some stuff to help me track this down some more if you can recompile your ompi.

Note, tomorrow and Friday are holiday's here in the U.S. so I won't probably get to responding to any email until Monday after today.

Terry D. Dontje | Principal Software Engineer

Developer Tools Engineering | +1.781.442.2631
Oracle - Performance Technologies
95 Network Drive, Burlington, MA 01803
Email terry.dontje@oracle.com