Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

From: Jeff Squyres (jsquyres_at_[hidden])
Date: 2007-10-12 11:31:40


On Oct 12, 2007, at 8:38 AM, Neeraj Chourasia wrote:

> Yes, the buffer was being re-used. No we didnt try to benchmark it
> with netpipe and other stuffs. But the program was pretty simple.
> Do you think, I need to test it with bigger chunks (>8MB) for
> communication.?
> We also tried manipulating eager_limit and min_rdma_sze, but no
> success.

It really depends on exactly what your application is doing. If you
re-use your buffers, then mpi_leave_pinned should help at least
some. But if the cost of communication is dwarfed by the rest of the
app, then all this tuning won't make much of a difference.

Here's a picture of Open MPI's MCA parameters w.r.t. the openib BTL
and the ob1 PML (we changed some names and ordering between the v1.2
series and what will become the v1.3 series -- both are shown below)
-- hopefully this will help you understand what those MCA parameters
are for:

The meanings of the 3 phases are explained in this pager: http://
www.open-mpi.org/papers/euro-pvmmpi-2006-hpc-protocols.

If you use the mpi_leave_pinned parameter and Open MPI is able to
leave your entire buffer registered, it should give the best
performance in OMPI v1.2.4 because it should be able to do a single
RDMA PUT to transfer the majority of the message (after the first
time, of course).

-- 
Jeff Squyres
Cisco Systems