There has been a bit of discussion about this on the core developers list
and on telecons, but I felt that perhaps I should provide a more detailed
warning to the broader developer community.
In the next few weeks, there will be some major revisions submitted to the
Open MPI trunk on the OpenRTE (ORTE) side of the code base. These will
primarily address three known issues:
1. Scalability - the test code that was run on Sandia's Thunderbird cluster
a few weeks ago utilized a stage gate and trigger to help speed up launch of
the OpenRTE daemons on backend nodes. In addition, some code cleanup
occurred in the TM launcher. These improvements yielded a positive result,
and they will be brought over to the trunk with these changes.
2. MAD (MPI_Abort Disease) - we have encountered a problem whereby daemons
are left "spinning" wildly when MPI processes call MPI_Abort. This is
symptomatic of a circular logic loop that has crept into the abort handling
section of the OpenRTE code base. These changes will resolve that problem.
3. Daemon timeout on start - currently, we will wait forever for all daemons
to start because we have no way to detect that they failed in some
environments. We are adding a timeout mechanism (adjustable via MCA param,
of course) that will allow orte/mpirun to give up after some period of time.
As part of these revisions, I am working to bring the code base another step
closer to OpenRTE 2.0 compatibility. As a result, some of the changes may
appear unnecessary in terms of fixing the three issues noted above. I
apologize in advance for that, but beg your indulgence as these changes will
make eventual integration with 2.0 a little easier.
The upcoming revisions will involve changes to the RDS, RAS, PLS, ERRMGR,
RMGR, and SMR frameworks in the form of API changes. Most of these changes
are not massive, but impact a number of places in the code. However,
significant change will occur in several places:
1. the ERRMGR will see significant change in actual behavior as we clarify
its role. New components to differentiate behavior between head node
processes (HNPs), daemons (our orteds), and application processes are being
2. communications to the OpenRTE daemons (orted's) will no longer take place
via individual frameworks but will be concentrated through the existing
orted non-blocking receive function. This will help us break the circular
logic loop and (hopefully) avoid re-creating it in the future.
3. the PLS "fork" component really was the orted's private launcher for
local processes. It has been moved to the orted's directory and renamed to
indicate that fact. Although there were good reasons to do this before, it
could not previously be done due to the built-in calls to the PLS - however,
with the new clarification of roles, this can now be cleanly done.
4. ALL resource management functionality has been constrained to the HNP.
Non-HNP processes (including orteds and application processes) solely
communicate their requests back to the HNP for execution. In addition, in
accordance with the OpenRTE 2.0 design, all resource management frameworks
(i.e., RDS, RAS, RMAPS, and PLS) are now publicly available (i.e., not just
through the RMGR).
5. the RMAPS framework has been changed to support multiple concurrent
mapping components, and a parameter added to the "map" API so the caller can
specify which one should be used for this specific map command.
For those of you with components in the affected frameworks, I am going
through them and making changes to keep them compatible with the revisions.
Again, these aren't major, but will require some checking to ensure they are
correct, especially for those components that will not compile unless in a
I hope to complete this work in the next 2-3 weeks. The work is taking place
on the tmp/mad branch - those of you with access to it are welcome to keep
track of what I am doing.
Prior to committing this massive a change to the trunk, I will be performing
testing on various platforms. I will also be contacting key people with
access to platforms beyond my domain to ask their help in testing the branch
in those environments.
And yes - I WILL send out a note alerting people to the upcoming commit
prior to throwing it into the trunk!
Feel free to contact me with any comments, suggestions, or concerns.