Open MPI logo

Open MPI Development Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Development mailing list

Subject: [OMPI devel] RFC: move BTLs out of ompi into separate layer
From: Rainer Keller (keller_at_[hidden])
Date: 2009-03-09 14:51:41


What: Move BTLs into separate layer

Why: Several projects have expressed interest to use the BTLs. Use-cases
such as the RTE using the BTLs for modex or tools collecting/distributing data
in the fastest possible way may be possible.

Where: This would affect several components, that the BTLs depend on
(namely allocator, mpool, rcache and the common part of the BTLs).
Additionally some changes to classes were/are necessary.

When: Preferably 1.5 (in case we use the Feature/Stable Release cycle ;-)

Timeout: 23.03.2009
------------------------------------------------------------------------

There has been much speculation about this project.
This RFC should shed some light, if there is some more information required,
please feel free to ask/comment. Of course, suggestions are welcome!

The BTLs offer access to fast communication framework. Several projects have
expressed interest to use them separate of other layers of Open MPI.
Additionally (with further changes) BTLs maybe used within ORTE itself.

COURSE OF WORK:
The extraction is not easy (as was the extraction of ORTE and OMPI in the
early stages of Open MPI?).
In order to get as much input and be as visible as possible (e.g. in TRACS),
the tmp-branch for this work has been set up on:
   https://svn.open-mpi.org/svn/ompi/tmp/koenig-btl

We propose to have a separate ONET library living in onet, based on orte (see
attached fig).

In order to keep the diff between the trunk and the branch to a minimum
several cleanup patches have already been applied to the trunk (e.g.
unnecessary #include of ompi and orte header files, integration of
ompi_bitmap_t into opal_bitmap_t, #include "*_config.h").

Additionally a script (attached below) has been kept up-to-date (contrib/move-
btl-into-onet), that will perform this separation on a fresh checkout of
trunk:
  svn list https://svn.open-mpi.org/svn/ompi/tmp/koenig-btl/contrib/move-btl-
into-onet

This script requires several patches (see attached TAR-ball).
Please update the variable PATCH_DIR to match the location of patches.

  ./move-btl-into-onet ompi-clean/
  # Lots of output deleted.
  cd ompi-clean/
  rm -fr ompi/mca/common/ # No two mcas called common, too bad...
  ./autogen.sh

OTHER RTEs:
A preliminary header file is provided in onet/include/rte.h to accommodate the
requirements of other RTEs (such as stci), that replaces selected
functionality, as proposed by Jeff and Ralph in the Louisville meeting.
Additionally, this header file is included before orte-header files (within
onet)...
By default, this does not change anything in the standard case (ORTE),
otherwise -DHAVE_STCI, redefinitions for components orte-functionality
required within onet is done.

TESTS:
First tests have been done locally on Linux/x86_64.
The branch compiles without warnings.
The wrappers have been updated.

The Intel Testsuite runs without failures:
  ./run-tests.pl all_tests_no_perf

PERFORMANCE:
!!!Before any merge, do extensive performance tests on real machines!!!
Initial tests on the cluster smoky, show no difference in comparison to ompi-
trunk.
Please see the enclosed output of NetPipe-3.7.1 run on a single node (--mca
btl sm,self) on smoky.

TODOS:
There are still some todos, to finalize this:
- Dependencies in the onet-layer into the ompi-layer (ompi_proc_t,
ompi_converter)
  We are working on these, and have shortly talked about the latter with
George.
- Better abstraction from orte / cleanups, such as modex

If these involve code-changes (and not just "save" and non-intrusive renames),
such as a opal_keyval-change, we will continue to write RFCs.

-- 
------------------------------------------------------------------------
Rainer Keller, PhD                  Tel: +1 (865) 241-6293
Oak Ridge National Lab          Fax: +1 (865) 241-4811
PO Box 2008 MS 6164           Email: keller_at_[hidden]
Oak Ridge, TN 37831-2008    AIM/Skype: rusraink