Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

From: Jeff Squyres (jsquyres_at_[hidden])
Date: 2007-06-11 08:17:32


Measuring communications is a very tricky process; there's a lot of
factors involved. Check out this FAQ item:

     http://www.open-mpi.org/faq/?category=tuning#running-perf-numbers

You might want to use a well-known benchmark program (e.g., NetPIPE,
link checker, etc.) to run pair-wise communication performance
analysis rather than write your own application; it's typically not
as simple as just doing a few sends within a loop.

The issue is that MPI may make different decisions on how to send
messages, including factors such as:

- is this the first time you have sent between these peer pair?
- who are you sending to?
- what is the size of the message?
- are there other messages pending?
- are other messages incoming from different peers while you are
sending?

Your simplistic loop below can cause some "bad" things to happen
(i.e., not give a true/absolute measure of what max performance is
between a pair of peers) by unintentionally stepping on several of
the things that Open MPI does behind the scenes (e.g., we don't make
network connections until the first time a message is sent between a
given peer pair).

But on the flip side, there's a whole school of thought that micro
benchmarks are only useful in a limited sense (because they test
artificial scenarios), and the only thing that *really* matters is
your application's performance. Hence, micro benchmarks are good as
input for guiding tuning issues, but they are not the absolute
measure of how well a given OS/middleware/network are performing.
That being said, a poorly-written application will tend perform
poorly regardless of how well the OS/middleware/network performs.

And so on.

This is an age-old religious debate, and both sides have some good
points. I won't re-hash the entire debate here. :-)

On Jun 4, 2007, at 10:00 AM, Allan, Mark ((UK Filton)) wrote:

> Hi,
>
> I'm new to this list and wonder if anyone can help. I'm trying to
> measure communication time between parallel processes using
> openmpi. As an example I might be running on 4 dual core
> processors (8 processes in total). I was hoping that communication
> using shared memory (comms between dual cores on the same chip)
> would be faster than that over the network. To measure
> communication time I'm sending a block of data to each process
> (from each process) using a blocking send, and am timing how long
> it takes. I repeat this 50 times (for example) and take the
> average time. The code is something like:
>
> for(int i=0;i<numProcs;i++)
> for(int j=0;j<numProcs;j++)
> if(i!=j)
> {
> // // // i is the sending proc to j, others wait
> double time = 0.0;
> for(int kk=0; kk<50; kk++)
> {
> if(i==my_rank)
> {
> double start = MPI::Wtime();
> MPI::COMM_WORLD.Send(&sendData
> [0],dataSize,MPI::DOUBLE,j,i);
> double end = MPI::Wtime();
> time+=(end-start);
> }
> if(j==my_rank)
> {
> MPI::COMM_WORLD.Recv(&recvData
> [0],dataSize,MPI::DOUBLE,i,i);
> }
> }
> if(i==my_rank)
> out << i << " " << j << " " << time/50.0 <<
> std::endl;
> MPI::COMM_WORLD.Barrier();
> }
>
> The problem I am having is that I'm not noticing any appreciable
> difference in communication times between shared memory and network
> protocols. I expected shared memory to be faster(!?!).
>
> Does anyone have a better way of measuring communication times?
>
> Thanks,
>
> Mark.
> ********************************************************************
> This email and any attachments are confidential to the intended
> recipient and may also be privileged. If you are not the intended
> recipient please delete it from your system and notify the sender.
> You should not copy it or use it for any purpose nor disclose or
> distribute its contents to any other person.
> ********************************************************************
>
> _______________________________________________
> users mailing list
> users_at_[hidden]
> http://www.open-mpi.org/mailman/listinfo.cgi/users

-- 
Jeff Squyres
Cisco Systems