One of the interesting aspects of an ABI MPI is that some folks won't
be convinced that it's interesting until they've had it proven that it
will work, both technically and socially, which is putting the cart
before the horse to a certain extent. It's also not helpful that all I
distributed were power-point slides; I did have a detailed writeup,
but it gave proposed solutions to too many problems, which is likely
to cause more arguments than it would solve.
In any case, discussion is a good thing, and here are some answers to
the issues you raise. I'm afraid that a casual observer won't
necessarily learn much from our exchange about whether or not an ABI
is possible or desirable. I also don't think I have all the answers,
but a few of the issues you raise do delve into things that I've
worked on in the past.
1. An ABI still leaves you vulnerable to different C++ and F90 issues.
On AMD64/EM64T and x86, Intel, gcc, and PathScale have compatible C++
implementations. This is not a coincidence.
The F90 issue is real, but note that it reduces to N libraries for
N compilers, instead of N*M libraries for N compilers and M MPIs.
1a. You're vulnerable to OS issues.
Yes. People who distribute binaries today already deal somewhat
successfully with that. We distribute only 4 sets of compiler RPMs,
but they happen to work successfully on RedHat Enterprise, Fedora, and
9, SUSE Professional and Enterprise, Mandrake, Debian, Gentoo, White
Hat, Rocks, and a few others I've forgotten because they're too
2. Choosing an MPI implementation
One existing solution is "modules". It takes the guesswork out of
setting and resetting PATH. It's definitely something that has to be
scripted or users can't use it reliably. You are correct that setting
LD_LIBRARY_PATH in an MPI job is an issue, but this needs a fix
anyway, as users run up against it all the time -- "Your Fortran docs
say I need to set this environment variable to change the output
format; it works with a serial code, but doesn't work with MPI! Help!"
3. This isn't an immediately perfect solution for all ISVs
Well, yes. I actually said that in my slides. For the average ISV,
being able to test other MPIs without recompiling is a first step
towards being able to eventually support more than one. But it's not
the only step. An above-average ISV will then be willing to let
customers use other MPIs, as long as any bugs are replicated using a
supported platform (i.e. using OpenMPI over TCP/IP). I expect it to
take a while for the average ISV to get there, but at least they have
a chance. Today they have no chance.
You also mention that some ISVs tend to distribute static linked
executables. This is true. Were I an ISV, I would distribute such an
executable for debugging purposes.
4. Non-standardized parts of MPI are an issue in real programs.
I agree with this one. I think these issues should be discussed as
part of the ABI effort. What parts are the worst problems, and can we
successfully standardize them?
5. Biologists and Chemists don't care about an ABI.
Yes, but they do care about having their apps just work. And an ABI
helps that. I can imagine a future in which an rpm for a biology app
distributes a test suite, and MPI implementers / interconnect vendors
who care will download the rpm and test it. That's easier with an ABI
than without; with an ABI you don't have to rebuild (which, for a
poorly packaged app, can be very tricky for a 3rd party to do), and
you can be fairly sure that you didn't introduce/delete a bug by
6. ISVs can ship MPI-independent apps today using a thin abstraction
Yes, but that wastes money and complicates testing. I wouldn't be
surprised if few of them did that. I suspect the ISVs that did that,
if any, would like an ABI more than their current system.
7. It's expensive to change sizes and types of MPI handles to a new
Look at your f77 interface -- it's a roadmap for converting from
fortran integer arguments to pointers. This is what I suggest you do
for C/C++. I'd be happy to write the code for you, actually.
8. Embedded MPI might have different constraints
An ABI doesn't have to be all things to all people. I find that most
HPC embedded systems are based on a fairly general-purpose platform,
so an ABI may manage to reach most of today's embedded players. (We
do do business in that space.)
9. Is recompiling really that difficult?
I don't think you or I are the right people to answer that one, we
recompile all the time. But I assure you that the compiler people in
my company would answer YES! YES! Why? Because compilers are heavily
QAed, and we can name numerous source files in our compiler that have
obscure bugs appear if compiled with gcc -O2 instead of -O0. These
bugs come and go, too, as we rearrange our source code. So for them,
they would never send a compiler binary out the door that had not been
fully QAed -- that exact binary. No "let's recompile for one fix and
then not re-run the entire test suite."
10. OpenMPI implements multiple batch queue systems already.
I did not mean to imply that no one had did this. In fact, several
MPIs have done it. But that doesn't help the average cluster admin,
who only has 1 batch queue system.
Likewise, many "portable" MPIs support multiple interconnects. But
that doesn't solve the issues associated with multiple interconnects,
for many reasons that have been stated already.
I do find it entertaining that you say supporting several run-times
is not difficult, while changing to support one mpi.h is expensive ;-)
11. Uber-mpirun sounds hard
I have a strawman idea for it; that's the kind of thing the committee
should discuss, not to be hashed out completely before the committee
ever meets. I've read many MPI startups and the strawman addresses all
of them. Quadrics and Portals both work with the concept.
I wouldn't be surprised if Uber-mpirun didn't make it into the ABI.
I'm OK with that, I don't think leaving it out fatally weakens the ABI
12. MPI-2 dynamic functions are hard to standardize
Yes, and lack of integration with queue systems is why I've never
observed a customer using them, ever. A quality-of-implementation
consensus would help these functions become useful, and is currently
crying out for action.
> Finally, this uber-mpirun will have a consistent syntax across all
> platforms and RTEs, but what about mpiexec? The MPI Forum explicitly
> specified mpiexec to fulfill this requirement. Has it failed? Are all
> the mpiexec implementations out there so radically different as to be
> useless in terms of uniform syntax? (this is an honest question)
I don't know, I've never used it. The fact that it isn't mentioned in
the slides is simple ignorance on my part ;-) One would hope that mpiexec
was complete enough, but it would be an interesting exercise to see
if, for the purposes of this ABI, it was or not.
> Having an MPI ABI gains nothing for MPI
> implementation researchers except that they don't have to recompile
> applications for their new implementation. This is exactly the same as
> it is for everyone else (per restrictions discussed above); singling
> out MPI implementation researchers is misleading.
I'm sorry if you find it misleading. I think it is useful to
specifically call out the benefit to an extremely important group of
people, the people for whom an MPI ABI is an expense. You are in that
group, and I think an ABI won't get going if you can't be convinced.
Hence I write 232 line emails...
> - On the "Winners: Interconnect implementors" slide: Why will
> interconnect implementors only reach systems that recompile?
I was referring to recompiling *applications*, not the MPI
implementation. Quadrics isn't supported by most of the "portable"
MPIs today. So a Quadrics customer has to compile applications
> - On the "Winners: Commercial software vendors" slide: I talked about
> this above. An ABI does *not* make testing easier --
It does, because you don't have to recompile, thus you can be confident
that you don't have new compiler issues, you'll only have MPI issues.
> Just because they don't have to recompile will not significantly reduce
> the logistics of all ISV's.
An ABI doesn't have to significantly help all ISVs to be successful.
> I don't see how automated testing becomes easier with an ABI.
If I don't have to recompile my ten million line application for each
interconnect, that's easier. I would suspect it cuts the work in half,
actually. Inside PathScale, for example, we build all 4 platforms
twice a day for each active branch, and archive *all* the builds, so
we can binary search for regressions. That's our minimum effort for "a
> - On the "Winners: Open Source Software Projects" slide: you say
> "Tomorrow, MPI is just like everything else..." Are you saying that
> MPI will be DLL Hell just like all other packages out there?
No, that it will mostly work, like most things today. DLL hell in
Linux, from my experience, happens when you want to use an RPM that's
too wildly different, or that depends on multiple not-commonly-
distributed libraries. The typical biologist downloads a set of RPMs
built for nearly-exactly the system they're on; they stay relatively
up to date on Linux distros because the people building the RPMs tell
them to do that.
> - On the "Issues: Startup and queue systems" slide: it sounds like you
> are now talking about standardizing queue systems which is a much, much
> larger effort than just the MPI (or even the HPC) community.
No, that was not my intent. I was talking about how uber-mpirun
interfaces, in a generic way, to queue systems. I know that
standardizing anything about queue systems is bound to fail, I've
personally seen a couple of such efforts not accomplish anything.
That's one thing I like about the MPI process, it has proven to be a
big success so far.
To sum up, if such a huge scatter-brained email can be summed up, I
think the cart is a bit before the horse in this discussion, and that
it would pay for us to listen to some of the people with bigger
benefit from an ABI before deciding it isn't interesting. After all,
MPI implementers pay the main cost of implementing an ABI, so it's no
surprise they're the hardest to convince. But, I think that this
discussion is an important step towards convincing people, hopefully