Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: Re: [OMPI users] Performance question about OpenMPI and MVAPICH2 on IB
From: Craig Tierney (Craig.Tierney_at_[hidden])
Date: 2009-08-06 19:03:08


A followup....

Part of problem was affinity. I had written a script to do processor
and memory affinity (which works fine with MVAPICH2). It is an
idea that I got from TACC. However, the script didn't seem to
work correctly with OpenMPI (or I still have bugs).

Setting --mca mpi_paffinity_alone 1 made things better. However,
the performance is still not as good:

Cores Mvapich2 Openmpi
---------------------------
   8 17.3 17.3
  16 31.7 31.5
  32 62.9 62.8
  64 110.8 108.0
 128 219.2 201.4
 256 384.5 342.7
 512 687.2 537.6

The performance number is GFlops (so larger is better).

The first few numbers show that the executable is the right
speed. I verified that IB is being used by using OMB and
checking latency and bandwidth. Those numbers are what I
expect (3GB/s, 1.5mu/s for QDR).

However, the Openmpi version is not scaling as well. Any
ideas on why that might be the case?

Thanks,
Craig

Craig Tierney wrote:
> I am running openmpi-1.3.3 on my cluster which is using
> OFED-1.4.1 for Infiniband support. I am comparing performance
> between this version of OpenMPI and Mvapich2, and seeing a
> very large difference in performance.
>
> The code I am testing is WRF v3.0.1. I am running the
> 12km benchmark.
>
> The two builds are the exact same codes and configuration
> files. All I did different was use modules to switch versions
> of MPI, and recompiled the code.
>
> Performance:
>
> Cores Mvapich2 Openmpi
> ---------------------------
> 8 17.3 13.9
> 16 31.7 25.9
> 32 62.9 51.6
> 64 110.8 92.8
> 128 219.2 189.4
> 256 384.5 317.8
> 512 687.2 516.7
>
> The performance number is GFlops (so larger is better).
>
> I am calling openmpi as:
>
> /opt/openmpi/1.3.3-intel/bin/mpirun --mca plm_rsh_disable_qrsh 1 --mca btl openib,sm,self \
> -machinefile /tmp/6026489.1.qntest.q/machines -x LD_LIBRARY_PATH -np $NSLOTS /home/ctierney/bin/noaa_affinity ./wrf.exe
>
> So,
>
> Is this expected? Are some common sense optimizations to use?
> Is there a way to verify that I am really using the IB? When
> I try:
>
> -mca bta ^tcp,openib,sm,self
>
> I get the errors:
> --------------------------------------------------------------------------
> No available btl components were found!
>
> This means that there are no components of this type installed on your
> system or all the components reported that they could not be used.
>
> This is a fatal error; your MPI process is likely to abort. Check the
> output of the "ompi_info" command and ensure that components of this
> type are available on your system. You may also wish to check the
> value of the "component_path" MCA parameter and ensure that it has at
> least one directory that contains valid MCA components.
> --------------------------------------------------------------------------
>
> But ompi_info is telling me that I have openib support:
>
> MCA btl: openib (MCA v2.0, API v2.0, Component v1.3.3)
>
> Note, I did rebuild OFED and put it in a different directory
> and did not rebuild OpenMPI. However, since ompi_info isn't
> complaining and the libraries are available, I am thinking that
> is isn't a problem. I could be wrong.
>
> Thanks,
> Craig

-- 
Craig Tierney (craig.tierney_at_[hidden])