Mathieu Gontier wrote:
> Dear OpenMPI users
> I am dealing with an arithmetic problem. In fact, I have two variants
> of my code: one in single precision, one in double precision. When I
> compare the two executable built with MPICH, one can observed an
> expected difference of performance: 115.7-sec in single precision
> against 178.68-sec in double precision (+54%).
> The thing is, when I use OpenMPI, the difference is really bigger:
> 238.5-sec in single precision against 403.19-sec double precision (+69%).
> Our experiences have already shown OpenMPI is less efficient than
> MPICH on Ethernet with a small number of processes. This explain the
> differences between the first set of results with MPICH and the second
> set with OpenMPI. (But if someone have more information about that or
> even a solution, I am of course interested.)
> But, using OpenMPI increases the difference between the two
> arithmetic. Is it the accentuation of the OpenMPI+Ethernet loss of
> performance, is it another issue into OpenMPI or is there any option a
> can use?
It is also unusual that the performance difference between MPICH and
OMPI is so large. You say that OMPI is slower than MPICH even at small
process counts. Can you confirm that this is because MPI calls are
slower? Some of the biggest performance differences I've seen between
MPI implementations had nothing to do with the performance of MPI calls
at all. It had to do with process binding or other factors that
impacted the computational (non-MPI) performance of the code. The
performance of MPI calls was basically irrelevant.
In this particular case, I'm not convinced since neither OMPI nor MPICH
binds processes by default.
Still, can you do some basic performance profiling to confirm what
aspect of your application is consuming so much time? Is it a
particular MPI call? If your application is spending almost all of its
time in MPI calls, do you have some way of judging whether the faster
performance is acceptable? That is, is 238 secs acceptable and 403 secs
slow? Or, are both timings unacceptable -- e.g., the code "should" be
running in about 30 secs.