Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: Re: [OMPI users] scaling problem with openmpi
From: Peter Kjellstrom (cap_at_[hidden])
Date: 2009-05-19 09:29:41


On Tuesday 19 May 2009, Roman Martonak wrote:
...
> openmpi-1.3.2 time per one MD step is 3.66 s
> ELAPSED TIME : 0 HOURS 1 MINUTES 25.90 SECONDS
> = ALL TO ALL COMM 102033. BYTES 4221. =
> = ALL TO ALL COMM 7.802 MB/S 55.200 SEC =
...
> mvapich-1.1.0 time per one MD step is 2.55 s
> ELAPSED TIME : 0 HOURS 1 MINUTES 0.65 SECONDS
> = ALL TO ALL COMM 102033. BYTES 4221. =
> = ALL TO ALL COMM 14.815 MB/S 29.070 SEC =
...
> Intel MPI 3.2.1.009 time per one MD step is 1.58 s
> ELAPSED TIME : 0 HOURS 0 MINUTES 38.16 SECONDS
> = ALL TO ALL COMM 102033. BYTES 4221. =
> = ALL TO ALL COMM 38.696 MB/S 11.130 SEC =
...
> Clearly the whole difference is basically in the ALL TO ALL COMM time.
> Running on 1 blade (8 cores) all three MPI implementations have very
> similar same time per step of about 8.6 s.

My guess is that what you see is the difference in MPI_Alltoall performance
for the different MPI-implementations (running in your env. on your hw.).

You could write a trivial loop like this and try on the three MPIs:

 MPI_init
 for i in 1 to 4221
   MPI_Alltoall(size=102033, ...)
 MPI_finialize

And time it to comfirm this.

> For CPMD I found that using the keyword TASKGROUP
> which introduces a different way of parallelization it is possible to
> improve on the openmpi time substantially and lower the time from 3.66
> s to 1.67 s, almost to the value found with Intel MPI.

I guess this changed what kind of communication is done and you no longer have
to do 4221x 100Kbytes alltoall that seems to hurt OpenMPI so much.

/Peter