Date: Thu, 12 Nov 2009 11:18:24 -0500
From: Gus Correa <gus@ldeo.columbia.edu>
Subject: Re: [OMPI users] mpi functions are slow when first called and
       become normal afterwards
To: Open MPI Users <users@open-mpi.org>
Message-ID: <4AFC3550.10500@ldeo.columbia.edu>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed

Eugene Loh wrote:
> RightCFD wrote:
>>
>>     Date: Thu, 29 Oct 2009 15:45:06 -0400
>>     From: Brock Palen <brockp@umich.edu <mailto:brockp@umich.edu>>
>>     Subject: Re: [OMPI users] mpi functions are slow when first called and
>>            become normal afterwards
>>     To: Open MPI Users <users@open-mpi.org <mailto:users@open-mpi.org>>
>>     Message-ID: <890CC430-68B0-4307-8260-24A6FADAE319@umich.edu
>>     <mailto:890CC430-68B0-4307-8260-24A6FADAE319@umich.edu>>
>>     Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
>>
>>     > When MPI_Bcast and MPI_Reduce are called for the first time, they
>>     > are very slow. But after that, they run at normal and stable speed.
>>     > Is there anybody out there who have encountered such problem? If you
>>     > need any other information, please let me know and I'll provide.
>>     > Thanks in advance.
>>
>>     This is expected, and I think you can dig though the message archive
>>     to find the answer.  OMPI does not wireup all the communication at
>>     startup, thus the first time you communicate with a host the
>>     connection is made, but after that it is fast because it is already
>>     open.  This behavior is expected, and is needed for very large systems
>>     where you could run out of sockets for some types of communication
>>     with so many hosts.
>>
>>     Brock Palen
>>     www.umich.edu/~brockp <http://www.umich.edu/%7Ebrockp>
>>     Center for Advanced Computing
>>     brockp@umich.edu <mailto:brockp@umich.edu>
>>     (734)936-1985
>>
>>     Thanks for your reply. I am surprised to know this is an expected
>>     behavior of OMPI. I searched the archival but did not find many
>>     relevant messages. I am wondering why other users of OMPI do not
>>     complain this. Is there a way to avoid this when timing an MPI
>>     program?
>>
> An example of this is the NAS Parallel Benchmarks, which have been
> around nearly 20 years.  They:
>
> *) turn timers on after MPI_Init and off before MPI_Finalize
> *) execute at least one iteration before starting timers
>
> Even so, with at least one of the NPB tests and with at least one MPI
> implementation, I've seen more than one iteration needed to warm things
> up.  That is, if you timed each iteration, you could see that multiple
> iterations were needed to warm everything up.  In performance analysis,
> it is reasonably common to expect to have to run multiple iterations and
> correct data set size to get representative behavior.
>
>

And I would guess in OpenMPI, maybe in other implementations too,
the time you spend warming up, probing the best way to do things,
is widely compensated for during steady state execution,
if the number of iterations is not very small.
This seems to be required to accommodate for the large variety
of hardware and software platforms, and be efficient on all of them.
Right?

AFAIK, other high quality software (e.g. FFTW)
do follow a similar rationale.

Gus Correa

> ------------------------------------------------------------------------
>
> _______________________________________________
> users mailing list
> users@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users



------------------------------

_______________________________________________
users mailing list
users@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users

End of users Digest, Vol 1403, Issue 4
**************************************