Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: Re: [OMPI users] Loopback Communication
From: Jeff Squyres (jsquyres_at_[hidden])
Date: 2008-03-01 08:20:35

On Feb 29, 2008, at 6:25 PM, Elvedin Trnjanin wrote:

> I'm using a "ping pong" program to approximate bandwidth and latency
> of
> various messages sizes and I notice when doing various transfers (eg.
> async) that the maximum bandwidth isn't the system's maximum
> bandwidth.
> I've looked through the FAQ and I haven't noticed this being covered
> but
> how does OpenMPI handle loopback communication? Is it still over a
> network interconnect or some sort of shared memory copy?

There are two kinds of loopback:

1. messages exchanged between two MPI processes on the same host.
This can be handled by most of OMPI's devices, but the best/fastest is
usually shared memory (i.e., the "sm" BTL).

2. messages exchanges between a single MPI process. This is handled
by the "self" OMPI device because it's just a memcpy within a single

So you'd typically want to run (assuming you have an IB network):

     mpirun --mca btl openib,self,sm ....

That being said, OMPI should usually pick the relevant BTL modules for
you (to include self and sm).

Jeff Squyres
Cisco Systems