Open MPI logo

Open MPI Development Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Development mailing list

Subject: [OMPI devel] RFC: openib BTL failover
From: Rolf vandeVaart (rolf.vandevaart_at_[hidden])
Date: 2010-06-02 17:22:00


WHAT: New PML called "bfo" (Btl Fail Over) that supports failover between
two or more openib BTLs. New configurable code in openib BTL that works
with the bfo to do failover. Note this only works when we have two or more
openib BTLs. This does not failover to another BTL, like tcp.

TO CONFIGURE:
--enable-openib-failover

TO RUN:
--mca pml bfo

TIMEOUT:
June 16, 2010

ADDITIONAL DETAILS:
The design relies on the BTL to call back into the PML with each
fragment that fails so the PML can decide what needs to be done.
There is no additional message tracking or software acknowledges
added so that we can have minimal impact on latency. Testing has
shown no measurable affect.

When errors are detected on the BTL, it is no longer used. No effort
is made to bring it back if the problems get corrected. If it gets
fixed before the next job starts, then it will be used by the next
job.

Under normal conditions, these changes have no effect whatsover on the
trunk as the bfo PML is never selected, and the failover support is
not configured into the openib BTL. Every effort was made to minimize
the changes in the openib BTL. The main changes are contained in two
new files that only get compiled when the -enable-openib-failover flag
is set. The other changes consist of about 75 new lines in various
openib BTL files.

The bitbucket version is at:
http://bitbucket.org/rolfv/rfc-failover

Here are the files that would be added/changed.

BTL LAYER
M ompi/mca/btl/btl.h
M ompi/mca/btl/base/btl_base_mca.c
M ompi/mca/btl/openib/btl_openib_component.c
M ompi/mca/btl/openib/btl_openib.c
M ompi/mca/btl/openib/btl_openib.h
M ompi/mca/btl/openib/btl_openib_endpoint.h
M ompi/mca/btl/openib/btl_openib_mca.c
A ompi/mca/btl/openib/btl_openib_failover.c
A ompi/mca/btl/openib/btl_openib_failover.h
M ompi/mca/btl/openib/btl_openib_frag.h
M ompi/mca/btl/openib/Makefile.am
M ompi/config/ompi_check_openib.m4

PML LAYER
A ompi/mca/pml/bfo
A ompi/mca/pml/bfo/pml_bfo_comm.h
A ompi/mca/pml/bfo/pml_bfo_sendreq.c
A ompi/mca/pml/bfo/pml_bfo_isend.c
A ompi/mca/pml/bfo/pml_bfo_component.c
A ompi/mca/pml/bfo/Makefile.in
A ompi/mca/pml/bfo/help-mpi-pml-bfo.txt
A ompi/mca/pml/bfo/pml_bfo_recvfrag.h
A ompi/mca/pml/bfo/pml_bfo_progress.c
A ompi/mca/pml/bfo/pml_bfo_sendreq.h
A ompi/mca/pml/bfo/pml_bfo_component.h
A ompi/mca/pml/bfo/pml_bfo_failover.c
A ompi/mca/pml/bfo/pml_bfo_recvreq.c
A ompi/mca/pml/bfo/pml_bfo_irecv.c
A ompi/mca/pml/bfo/pml_bfo_failover.h
A ompi/mca/pml/bfo/pml_bfo_recvreq.h
A ompi/mca/pml/bfo/pml_bfo_iprobe.c
A ompi/mca/pml/bfo/pml_bfo.c
A ompi/mca/pml/bfo/post_configure.sh
A ompi/mca/pml/bfo/pml_bfo_hdr.h
A ompi/mca/pml/bfo/pml_bfo_rdmafrag.c
A ompi/mca/pml/bfo/pml_bfo_rdma.c
A ompi/mca/pml/bfo/configure.params
A ompi/mca/pml/bfo/pml_bfo.h
A ompi/mca/pml/bfo/pml_bfo_rdmafrag.h
A ompi/mca/pml/bfo/pml_bfo_rdma.h
A ompi/mca/pml/bfo/.windows
A ompi/mca/pml/bfo/Makefile.am
A ompi/mca/pml/bfo/pml_bfo_comm.c
A ompi/mca/pml/bfo/pml_bfo_start.c
A ompi/mca/pml/bfo/pml_bfo_recvfrag.c