Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

From: Jeff Squyres (jsquyres_at_[hidden])
Date: 2005-06-16 09:30:00

On Jun 15, 2005, at 9:34 PM, Jonathan Day wrote:

> Phew! I'd no idea my questions would create nearly
> that kind of level of response on the list. I should
> post more often! :)

Please do. :-)

In all honesty, no one likes getting negative feedback, but hey, if
it's on the mark, then it's good feedback -- regardless of whether you
like it or not.

> Anyways, here are a few suggestions that might - just
> might - work around some of the problems that have
> been mentioned:
> 1. Publish the stable header files early, or fragments
> thereof
> People don't need the finished code to start
> developing against your API, all they really need is a
> list of prototypes. After all, we were all taught to
> design first and then implement, right? :)
> [snipped]

We actually considered doing this. However, we eventually decided
against it for the following reasons:

1. We actually did design first (yay software engineering!) --
top-level interface header files were written first (even including
doxygen comments!) and then implementations were written from that.
However, as time went on, those interfaces changed (sometimes wildly,
unfortunately) as we realized that we got things wrong in our initial
design. The run-time system is the best example of that -- it took us
a *long* time to figure out exactly what we wanted from OMPI's run-time
system, and as such, we effectively threw away the first few designs.

The point is that even the best designs change over time. That would
have been horrible -- to give a design to 3rd parties, and then have
them find out that it had been changed (and/or scrapped/restarted) a
few months later. This was one of the major rationale behind our
desire to "reach a level of stability" before releasing to the
community -- that there would be at least some expected level
un-changing-ness in our interfaces (I'm sure they won't be locked in
stone when we release, but we at least want to be fairly confident that
they're "good enough" and won't have major design changes down the

2. With an entirely new design, expecting a 3rd party to write solely
to a design is a difficult (at best) endeavor. Even with full, precise
documentation, there's always the issue of interpretation -- one
developer may mean X but someone else reads it as Y. These types of
things only tend to only show up once you have software that you can
actually compile and test against.

Hence, we really wanted to provide software along with the design that
could be used for development and testing.

> 2. Parallel Trees
> What I am thinking of here is something akin to Andrew
> Morton's or Alan Cox' patch series for Linux. You have
> the "official" Open-MPI, and then you have something
> that contains all kinds of bits and bobs that aren't
> stable, aren't validated, and may never be included in
> the official release, but which would still be of
> interest to researchers.

Actually, we have in mind something very much like this. Let me
explain a little background first.

The majority of Open MPI is implemented via components. So a *lot* of
Open MPI's implementation is selectable at run-time, meaning that you
can swap in/out different parts (including research-quality parts) at
will. The components are [usually] dynamic shared objects (DSOs), so
you can add/subtract .so's or DLLs after Open MPI has been installed.

I've cited several of Open MPI's goals in posts to this list over the
past 24 hours -- another of our goals is to provide a stable platform
for parallel / message passing research. Do you want to have an MPI
over a new message passing network architecture? Simply write a new
component. Do you want to try out some new whiz-bang collective
algorithms? Write a new component. ...and so on -- we currently have
over 20 component types in Open MPI, roughly divided between MPI and
the back-end run-time system.

This allows 3rd parties to explore the areas that they care about
without worrying about most of the other junk that you have to have for
a full MPI implementation. For example, I have seen collective
algorithm researchers write *an entire MPI implementation* just so that
they could play around with new algorithms.

That may be somewhat of an extreme example, but I think there should
never be a need for anything remotely like that. A proper framework
for an MPI implementation should enable 3rd parties to add / extend /
modify an existing MPI without nearly that much work.

So -- the point here is that the majority of Open MPI's functionality
is delivered through components. And you can substitute / add your own
functionality by writing your own components. In short: it's all about

But, as I stated yesterday, it's all about the HPC community. And some
of us are academics, and we know that in academia, it's all about
getting the credit for the stuff that you have done. How can we get
the community involved, yet still give credit (and blame ;-) ) to those
who deserve it?

There's actually already a community that has a very similar model --
Perl. The CPAN system is great for downloading newer versions of
components and/or esoteric components for lesser-known functionality.
The authors of each component are clearly denoted, so it's quite clear
who gets the credit/blame.

We plan to host a CPAN-like central repository (or, more to the point,
the root of a distributed repository) that allows people to post
notices of their components and/or download others components.
Ramifications of this:

1. if you write some cool Open MPI components, they do *not* need to be
shipped with Open MPI. We really hope to prevent the project-forking
problem that has become quite a problem for MPI implementations.

2. if you provide some cool Open MPI components, you can distribute
them separately from Open MPI, since they can be installed after the
fact. For example, you can put the components on your own web site
and register them with the central repository, thereby making them
available for download via a simple CPAN-like tool. **This even
applies to vendors.**

3. components can be distributed in source or binary form (read: as
open or closed source). We're open source [I know, I know], but we
recognize that not everyone else is.

4. components can be installed globally or locally (i.e., in the
central Open MPI installation tree or elsewhere, such as under an
individual user's $HOME), allowing development work to not interfere
with production runs.

Back to the repository model...

This all sounds like a Grand Master Plan (and it is!), but it's far
less work to implement than you would think. Other projects have done
similar things with command line / menu-driven tools implemented in
only a few hundred lines of perl. So although we haven't implemented
this yet, we're quite confident that it won't be difficult. The most
difficult part will be making pretty web pages to reflect this stuff
(we're engineers, meaning that we're horrid web designers).

All this being said, there are currently some limitations to Open MPI's
component scheme (e.g., there's only one set of MPI bindings -- they're
not component-ized, so to speak). We do plan on relaxing them over
time to make Open MPI be a framework of frameworks; i.e., making Open
MPI even more pluggable than it already is. However, until that time,
if you have changes to the non-component parts of Open MPI, we'll be
accepting contributions and patches the old-fashioned way.

Someday, in the hopefully-not-distant-future, we'll have 3rd party
contribution forms like the Apache foundation which will facilitate
this mechanism for those who want to be actively involved. But as I
mentioned yesterday, this is unfortunately taking far longer to setup
than we anticipated.

> 3. Own Your Code
> It is hard to prove ownership, in a digital age, but
> not completely impossible. For each function, you want
> to generate a SHA-1 hash of the source and of the
> binary (using default compiler options). You then do
> likewise for each source file and object file and then
> for each compiled program and library.

Not a bad idea -- thanks!

On a related note, the license on Open MPI will essentially be the BSD
license. We added one paragraph, but the spirit is the same -- you can
do whatever you want with the code as long as you preserve our
copyrights (academics read: we still get credit).

This is designed to be vendor-friendly, and even 3rd-party-researcher
friendly. So we don't care if you copy the code, just as long as you
properly cite us (I am not a lawyer and this is not legal advice -- if
you have questions, please read our LICENSE file and/or discuss it with
a lawyer).

I've attached our current LICENSE file for reference. It may change
slightly in the future to simplify the licensing because there are
several different copyright holders.

Hope this helps clarify even more issues! :-)

{+} Jeff Squyres
{+} The Open MPI Project

  • application/octet-stream attachment: LICENSE