Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

From: Jeff Squyres \(jsquyres\) (jsquyres_at_[hidden])
Date: 2006-05-26 07:18:41

Open MPI is based on a component architecture, where plugins are used
for various types of back-end functionality.

For MPI point-to-point communication, there are three types of plugins
that are used:

- pml: Point to Point Messaging Layer (what is directly invoked behind
MPI_SEND and friends). This layer understands MPI semantics.
- btl: Byte Transfer Layer (lowest layer "network driver" for
communicating across specific networks). This layer is completely
MPI-agnostic; it just pushes bytes between endpoints (usually
- bml: BTL Management Layer (for loading, unloading, and generally
run-time managing multiple BTL plugins).

Note that previous generations of Open MPI used "ptl" plugins instead of
"btl" plugins.

I would suggest reading the "Open MPI: A Flexible High Performance MPI"
paper on the Open MPI web site (see the "Publications" section) to get
an overview of the OB1 pml plugin. It describes the basis and top-level
design of how the pml, bml, and btls interact at run-time. Also read
"InfiniBand Scalability in Open MPI" (also in the publications section).
It is much more recent and, although somewhat IB-specific, also gives a
good overview of how things work. Finally, look at the MPI Developer's
Workshop slides. They probably provide the best description of what is
happening for point-to-point messaging (there's whole sections devoted
to this).

If you simply want to implement a new network driver (e.g., raw
ethernet), you will want to write a new BTL plugin. If you want to play
with various scheduling policies, you probably want to modify / write a
new PML plugin. This code is not well documented other than comments in
the code and the description of the plugin interfaces; your best bet is
to ask questions here on the mailing list.

I strongly recommend working with a Subversion checkout -- things have
changed since the 1.0 release. See for
details on getting a subversion checkout.

Within the code base, you'll be working within the ompi/ tree (the MPI
layer). You'll find the MPI C API bindings in ompi/mpi/c/*.c (e.g.,
MPI_SEND is in ompi/mpi/c/send.c). Also in ompi/ is the mca/ directory
which, in turn, has subdirectories for each kind of plugin (a.k.a.
"frameworks"), so you'll see subdirectories for pml/ and bml/ and btl/.
In each framework's directory are subdirectories for each plugin (a.k.a.
"components"). So you'll see ompi/mca/pml/ob1 (code for the ob1 PML
plugin), ompi/mca/bml/r2 (code for the r2 BML plugin), ompi/mca/btl/tcp
(code for the tcp BTL plugin), etc.

This is probably enough to get you started. Let us know what your next
questions are.

> -----Original Message-----
> From: users-bounces_at_[hidden]
> [mailto:users-bounces_at_[hidden]] On Behalf Of Mahesh Barve
> Sent: Tuesday, May 23, 2006 1:24 AM
> To: users_at_[hidden]
> Subject: [OMPI users] Help regarding send/recv code
> Hi,
> I am a novice to openmpi. Just managed to get openmpi
> running on my system.
> I would like to modify the code for send and recv.
> The target lower level device will be ethernet and
> infiniband. I would like to know the files/functions
> to look for. Could you please guide me in this.
> thanks,
> -Mahesh
> __________________________________________________
> Do You Yahoo!?
> Tired of spam? Yahoo! Mail has the best spam protection around
> _______________________________________________
> users mailing list
> users_at_[hidden]