Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: Re: [OMPI users] mpi functions are slow when first called and become normal afterwards
From: Eugene Loh (Eugene.Loh_at_[hidden])
Date: 2009-11-12 10:03:46


RightCFD wrote:
Date: Thu, 29 Oct 2009 15:45:06 -0400
From: Brock Palen <brockp@umich.edu>
Subject: Re: [OMPI users] mpi functions are slow when first called and
       become normal afterwards
To: Open MPI Users <users@open-mpi.org>
Message-ID: <890CC430-68B0-4307-8260-24A6FADAE319@umich.edu>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes

> When MPI_Bcast and MPI_Reduce are called for the first time, they
> are very slow. But after that, they run at normal and stable speed.
> Is there anybody out there who have encountered such problem? If you
> need any other information, please let me know and I'll provide.
> Thanks in advance.

This is expected, and I think you can dig though the message archive
to find the answer.  OMPI does not wireup all the communication at
startup, thus the first time you communicate with a host the
connection is made, but after that it is fast because it is already
open.  This behavior is expected, and is needed for very large systems
where you could run out of sockets for some types of communication
with so many hosts.

Brock Palen
www.umich.edu/~brockp
Center for Advanced Computing
brockp@umich.edu
(734)936-1985

Thanks for your reply. I am surprised to know this is an expected behavior of OMPI. I searched the archival but did not find many relevant messages. I am wondering why other users of OMPI do not complain this. Is there a way to avoid this when timing an MPI program?
An example of this is the NAS Parallel Benchmarks, which have been around nearly 20 years.  They:

*) turn timers on after MPI_Init and off before MPI_Finalize
*) execute at least one iteration before starting timers

Even so, with at least one of the NPB tests and with at least one MPI implementation, I've seen more than one iteration needed to warm things up.  That is, if you timed each iteration, you could see that multiple iterations were needed to warm everything up.  In performance analysis, it is reasonably common to expect to have to run multiple iterations and correct data set size to get representative behavior.