Thanks for all your inputs.
It is good to know this initial latency is an expected behavior and the
workaround by using one dummy iteration before timing is started.
I did not notice this because my older parallel CFD code runs a large number
of time steps and the initial latency was compensated.
But recently I am teaching MPI stuff using small parallel codes and noticed
This relieves my concern about our system performance.
> Date: Thu, 12 Nov 2009 11:18:24 -0500
> From: Gus Correa <gus_at_[hidden]>
> Subject: Re: [OMPI users] mpi functions are slow when first called and
> become normal afterwards
> To: Open MPI Users <users_at_[hidden]>
> Message-ID: <4AFC3550.10500_at_[hidden]>
> Content-Type: text/plain; charset=ISO-8859-1; format=flowed
> Eugene Loh wrote:
> > RightCFD wrote:
> >> Date: Thu, 29 Oct 2009 15:45:06 -0400
> >> From: Brock Palen <brockp_at_[hidden] <mailto:brockp_at_[hidden]>>
> >> Subject: Re: [OMPI users] mpi functions are slow when first called
> >> become normal afterwards
> >> To: Open MPI Users <users_at_[hidden] <mailto:users_at_[hidden]>>
> >> Message-ID: <890CC430-68B0-4307-8260-24A6FADAE319_at_[hidden]
> >> <mailto:890CC430-68B0-4307-8260-24A6FADAE319_at_[hidden]>>
> >> Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
> >> > When MPI_Bcast and MPI_Reduce are called for the first time, they
> >> > are very slow. But after that, they run at normal and stable
> >> > Is there anybody out there who have encountered such problem? If
> >> > need any other information, please let me know and I'll provide.
> >> > Thanks in advance.
> >> This is expected, and I think you can dig though the message archive
> >> to find the answer. OMPI does not wireup all the communication at
> >> startup, thus the first time you communicate with a host the
> >> connection is made, but after that it is fast because it is already
> >> open. This behavior is expected, and is needed for very large
> >> where you could run out of sockets for some types of communication
> >> with so many hosts.
> >> Brock Palen
> >> www.umich.edu/~brockp <http://www.umich.edu/%7Ebrockp>
> >> Center for Advanced Computing
> >> brockp_at_[hidden] <mailto:brockp_at_[hidden]>
> >> (734)936-1985
> >> Thanks for your reply. I am surprised to know this is an expected
> >> behavior of OMPI. I searched the archival but did not find many
> >> relevant messages. I am wondering why other users of OMPI do not
> >> complain this. Is there a way to avoid this when timing an MPI
> >> program?
> > An example of this is the NAS Parallel Benchmarks, which have been
> > around nearly 20 years. They:
> > *) turn timers on after MPI_Init and off before MPI_Finalize
> > *) execute at least one iteration before starting timers
> > Even so, with at least one of the NPB tests and with at least one MPI
> > implementation, I've seen more than one iteration needed to warm things
> > up. That is, if you timed each iteration, you could see that multiple
> > iterations were needed to warm everything up. In performance analysis,
> > it is reasonably common to expect to have to run multiple iterations and
> > correct data set size to get representative behavior.
> And I would guess in OpenMPI, maybe in other implementations too,
> the time you spend warming up, probing the best way to do things,
> is widely compensated for during steady state execution,
> if the number of iterations is not very small.
> This seems to be required to accommodate for the large variety
> of hardware and software platforms, and be efficient on all of them.
> AFAIK, other high quality software (e.g. FFTW)
> do follow a similar rationale.
> Gus Correa
> > ------------------------------------------------------------------------
> > _______________________________________________
> > users mailing list
> > users_at_[hidden]
> > http://www.open-mpi.org/mailman/listinfo.cgi/users
> users mailing list
> End of users Digest, Vol 1403, Issue 4