Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: Re: [OMPI users] Performance question about OpenMPI and MVAPICH2 on IB
From: Gus Correa (gus_at_[hidden])
Date: 2009-08-07 12:50:46


Hi Craig, Terry, Neeraj, list

Craig: A fellow here runs WRF.
I grep the code and there is plenty of collectives there:
MPI_[All]Gather[v], MPI_[All]Reduce, etc.
Domain decomposition code like WRF, MITgcm, and other atmosphere
and ocean codes has point-to-point communication to exchange
subdomain boundaries, but also collective operations to calculate
sums, etc, in various types of PDE (matrix) solvers that require
global information.

Terry: On the MITgcm, the apparent culprit is MPI_Allreduce,
which seems to be bad on **small** messages (rather than big ones).
This is the same behavior pattern that was reported here on May,
regarding MPI_Alltoall, by Roman Martonak, a list subscriber using a
computational chemistry package on an IB cluster:

http://www.open-mpi.org/community/lists/users/2009/07/10045.php
http://www.open-mpi.org/community/lists/users/2009/05/9419.php

At that point Pavel Shamis, Peter Kjellstrom, and others gave
very good suggestions, but they were only focused on MPI_Alltoall.
No other collectives were considered.

All: Any insights on how to tune MPI_Allreduce?
Maybe a hint on the other collectives also?
Any benchmark tool available that one can use to find the
sweet spot of each collective?

Many thanks,
Gus Correa
---------------------------------------------------------------------
Gustavo Correa
Lamont-Doherty Earth Observatory - Columbia University
Palisades, NY, 10964-8000 - USA
---------------------------------------------------------------------

neeraj_at_[hidden] wrote:
> Hi Terry,
>
> I had tested mostly MPI_Bcast, MPI_Reduce, MPI_Gather kind of
> MPI with openmpi-1.3 and hierarchical option enabled.In all these, i
> found results slower than regular tuned collectives.
>
> We have HP Blade with intel clovertown processor(two quad core)
> connected with DDR infiniband clos network.
> Results were tested on 12-16 nodes with 8 mpi process each node.
>
>
> Regards
>
> Neeraj Chourasia (MTS)
> Computational Research Laboratories Ltd.
> (A wholly Owned Subsidiary of TATA SONS Ltd)
> B-101, ICC Trade Towers, Senapati Bapat Road
> Pune 411016 (Mah) INDIA
> (O) +91-20-6620 9863 (Fax) +91-20-6620 9862
> M: +91.9225520634
>
>
>
> *Terry Dontje <Terry.Dontje_at_[hidden]>*
> Sent by: users-bounces_at_[hidden]
>
> 08/07/2009 05:15 PM
> Please respond to
> Open MPI Users <users_at_[hidden]>
>
>
>
> To
> users_at_[hidden]
> cc
>
> Subject
> Re: [OMPI users] Performance question about OpenMPI and MVAPICH2
> on IB
>
>
>
>
>
>
>
>
> Hi Neeraj,
>
> Were there specific collectives that were slower? Also what kind of
> cluster were you running on? How many nodes and cores per node?
>
> thanks,
>
> --td
> > Message: 3
> > Date: Fri, 7 Aug 2009 16:51:05 +0530
> > From: neeraj_at_[hidden]
> > Subject: Re: [OMPI users] Performance question about OpenMPI and
> > MVAPICH2 on IB
> > To: Open MPI Users <users_at_[hidden]>
> > Cc: users_at_[hidden], users-bounces_at_[hidden]
> > Message-ID:
> >
> <OF62A95E62.D6758124-ON6525760B.003E2874-6525760B.003E1A66_at_[hidden]>
> >
> > Content-Type: text/plain; charset="us-ascii"
> >
> > Hi Terry,
> >
> > I feel hierarchical collectives are slower compare to tuned
> one. I
> > had done some benchmark in the past specific to collectives, and this is
> > what i feel based on my observation.
> >
> > Regards
> >
> > Neeraj Chourasia (MTS)
> > Computational Research Laboratories Ltd.
> > (A wholly Owned Subsidiary of TATA SONS Ltd)
> > B-101, ICC Trade Towers, Senapati Bapat Road
> > Pune 411016 (Mah) INDIA
> > (O) +91-20-6620 9863 (Fax) +91-20-6620 9862
> > M: +91.9225520634
>
> _______________________________________________
> users mailing list
> users_at_[hidden]
> http://www.open-mpi.org/mailman/listinfo.cgi/users
>
> =====-----=====-----===== Notice: The information contained in this
> e-mail message and/or attachments to it may contain confidential or
> privileged information. If you are not the intended recipient, any
> dissemination, use, review, distribution, printing or copying of the
> information contained in this e-mail message and/or attachments to it
> are strictly prohibited. If you have received this communication in
> error, please notify us by reply e-mail or telephone and immediately and
> permanently delete the message and any attachments. Internet
> communications cannot be guaranteed to be timely, secure, error or
> virus-free. The sender does not accept liability for any errors or
> omissions.Thank you =====-----=====-----=====
>
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> users mailing list
> users_at_[hidden]
> http://www.open-mpi.org/mailman/listinfo.cgi/users