Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: Re: [OMPI users] Question about OpenMPI performance vs. MVAPICH2
From: Ralph Castain (rhc_at_[hidden])
Date: 2009-09-20 11:49:05


Did you set -mca mpi_paffinity_alone 1? This will bind the processes
to cores and (usually) significantly improve performance.

The upcoming 1.3.4 will have additional binding options to help with
performance issues.

On Sep 20, 2009, at 12:36 AM, Brian Powell wrote:

> Greetings,
>
> I recently purchased and set up a new blade cluster using Xeon 5560
> CPUs, Mellanox DDR ConnectX cards, running CentOS 5.2. I use the
> cluster to run a large FORTRAN 90 fluid model. I have been using
> OpenMPI on my other clusters for years, and it is my default MPI
> environment.
>
> I downloaded and installed the latest OpenMPI 1.3.3 release with the
> following:
>
> ./configure FC=ifort F77=ifort F90=ifort --prefix=/share/apps/
> openmpi-1.3.3-intel --with-openib=/opt/ofed --with-openib-libdir=/
> opt/ofed/lib64 --with-tm=/opt/torque/
>
> To show the configuration, I ran:
>
> (machine:~)% mpicc -v
> Using built-in specs.
> Target: x86_64-redhat-linux
> Configured with: ../configure --prefix=/usr --mandir=/usr/share/man
> --infodir=/usr/share/info --enable-shared --enable-threads=posix --
> enable-checking=release --with-system-zlib --enable-__cxa_atexit --
> disable-libunwind-exceptions --enable-libgcj-multifile --enable-
> languages=c,c++,objc,obj-c++,java,fortran,ada --enable-java-awt=gtk
> --disable-dssi --enable-plugin --with-java-home=/usr/lib/jvm/
> java-1.4.2-gcj-1.4.2.0/jre --with-cpu=generic --host=x86_64-redhat-
> linux
> Thread model: posix
> gcc version 4.1.2 20080704 (Red Hat 4.1.2-44)
>
> I then ran a large number of tests using one of my typical model
> domain configurations (which are relatively expensive) to test how
> well the system was performing. I didn't want to use "benchmarking"
> code, but rather the code I actually use the cluster for.
> Remarkably, it was scaling linearly up to about 8 nodes (using 8
> cores per node).
>
> I decided -- for curiosity -- to see how this compared with
> MVAPICH2. I downloaded the 1.4rc2 code, and compiled it using the
> following:
>
> ./configure FC=ifort F77=ifort F90=ifort --prefix=/share/apps/
> mvapich2-1.4-intel --enable-f90 --with-ib-libpath=/opt/ofed/lib64 --
> with-rdma=gen2 --with-ib-include=/opt/ofed/include
>
> This was confirmed with:
>
> (machine:~)% mpicc -v
> mpicc for 1.4.0rc2
> Using built-in specs.
> Target: x86_64-redhat-linux
> Configured with: ../configure --prefix=/usr --mandir=/usr/share/man
> --infodir=/usr/share/info --enable-shared --enable-threads=posix --
> enable-checking=release --with-system-zlib --enable-__cxa_atexit --
> disable-libunwind-exceptions --enable-libgcj-multifile --enable-
> languages=c,c++,objc,obj-c++,java,fortran,ada --enable-java-awt=gtk
> --disable-dssi --enable-plugin --with-java-home=/usr/lib/jvm/
> java-1.4.2-gcj-1.4.2.0/jre --with-cpu=generic --host=x86_64-redhat-
> linux
> Thread model: posix
> gcc version 4.1.2 20080704 (Red Hat 4.1.2-44)
>
> I tested the same runs as before, now using MVAPICH2 rather than
> OpenMPI. To my astonishment, the MVAPICH2 runs ran -- on average --
> 20% faster as measured in terms of wall clock time. This was
> incredibly surprising to me. I tried a number of domain
> configurations (over 1-16 nodes, with various numbers of processors
> per node), and the improvement was from 7.7-35.2 percent (depending
> on the configuration).
>
> I reran a number of my OpenMPI tests because it was so surprising,
> and they were consistent with the original. I read through the FAQ: <http://www.open-mpi.org/faq/?category=openfabrics
> > and tried a number of options with RDMA (the size of the messages
> passed in the code I run is -- I believe -- rather small) and I was
> able to improve the OpenMPI results by about 3%, but still nowhere
> near what I was getting with MVAPICH2.
>
> I ran a final test which I find very strange: I ran the same test
> case on 1 cpu. The MVAPICH2 case was 23% faster!?!? This makes
> little sense to me. Both are using ifort as the mpif90 compiler
> using *identical* optimization flags, etc. I don't understand how
> the results could be different.
>
> All of these cases are run with myself as the only user of the
> cluster and each test is run alone (without any other interference
> on the machine). I am running TORQUE, so each is submitted to the
> queue, then the actual queue run time is used as the measure of
> time, which is the actual wallclock time for the job to finish. Some
> may discount that time metric; however, it is what I am most
> concerned with. If I have to wait 2 hours to run a job in OpenMPI,
> but 1:36 in MVAPICH2, that is a significant advantage to me.
>
> That said, MVAPICH2 has its own problems with hung mpd processes
> that can linger around on the nodes, etc. I prefer to use OpenMPI,
> so my question is:
>
> What does the list suggest I modify in order to improve the OpenMPI
> performance?
>
> I have played with the RDMA parameters to increase its thresholds,
> but little was gained. I am happy to provide the output of ompi_info
> if needed, but it is long so I didn't want to include in the initial
> post. I apologize for my naivete on the internals of MPI hardware
> utilization.
>
> Thank you in advance.
>
> Cheers,
> Brian
>
>
>
>
>
> _______________________________________________
> users mailing list
> users_at_[hidden]
> http://www.open-mpi.org/mailman/listinfo.cgi/users