Hi all -
LANL had an internal meeting yesterday trying to classify a number of
issues we're having with the run-time environment for Open MPI and how
to best prioritize team resources. We thought it would be good to
both share the list (with priorities) with the group and to ask the
group if there were other issues that need to be addressed (either
short or long term). We've categorized the issues as performance
related, robustness, and feature / platform support. The numbers are
the current priority on our list, and items within a category are
sorted by priority.
PERFORMANCE:
5) 50% scale factor in process startup
Start-up of non-MPI jobs has a strange bend in the timing curve
when the number of processes we are trying to start is greater than
or equal to 50% of the current allocation. It appears that
starting a 16 process (1 ppn) job takes longer if there are 32
nodes in the allocation than if there are 64 nodes in the
allocation.
Assigned to: Galen
6) MPI_INIT startup timings
In addition to seeming to suffer from the same 50% issue as the
previous issue, there also appears to be a number of places in
MPI_INIT where we spend a considerable amount of time when at
scale, leading to startup times much worse than LA-MPI or
MPIEXEC/MVAPICH.
Assigned to: Galen
ROBUSTNESS:
1) MPI process aborting issue
This is the orted spin, MPI processes don't die, etc. issue that
occurs when some process dies unexpectedly. Ralph has already sent
a detailed e-mail to devel about this issue.
Assigned to: Ralph
1.5) MPI_ABORT rework
The MPI process aborting issue is going to require a rework of
MPI_ABORT so that it uses the error manager instead of calling
terminate_proc/terminate_job.
Assigned to: Brian
2) ORTE hangs when start-up fails
If an orted fails to start or fails to connect back to the HNP, the
system hangs waiting for the callback. If a orted process fails to
start entirely, we sometimes catch this. But we need a better
mechanism for handling the general failure case.
Assigned to: Ralph
3) Hardened cleanup of session directory
While #1 should greatly help in ensuring that the session directory
is cleaned up every time, there are still a number of race
conditions that need to be sorted out. The goal is to develop a
plan that ensures files that need to be removed are removed
automatically a high percentage of the time, that there is a way to
allow a tool like orte_clean to clean up everything it should clean
up, and that there is a way to make sure files that should not be
automatically removed aren't automatically removed.
Assigned to: Brian
3.5) Process not found hangs
See https://svn.open-mpi.org/trac/ompi/ticket/245
Assigned to: Ralph
7) Node death failures / hangs
With the exception of BProc, if a node fails, we don't detect the
failure. Even if we did detect the failure, we have no general
mechanism for dealing with that failure. The bulk of this project
is going to be adding a general SOH/SMR component that uses the OOB
for timeout pings.
Assigned to: Brian
15) More friendly error messages
There are situations where we give something south of a useful
error message when an error is found. We should play nicer with
users.
Assigned to:
16) Consistent error checking
We've had a number of recent instances of errors occuring, but not
being propogated / returned to the user simply because no one ever
checked the return code. We need to audit most of ORTE to always
check return codes.
Assigned to:
FEATURE / PLATFORM SUPPORT:
4) TM error handling
TM, while used on a number of large systems LANL needs to support,
is not exactly friendly to usage at scale. It seems that it likes
to go away and cry to mamma for a couple seconds, returning system
error messages, only to come back and be ok a second later. This
means that every TM call needs to be handled as if it's going to
fail, and we need to be prepared to re-initialize the system (if
possible) when failures occur. In testing on t-bird, launching was
usually pretty stable, but the calls to get the node allocations
tended to result in the strange behavior. These should definitely
be re-startable type errors
Assigned to: Brian
8) Hetergeneous Issues
Assigned to:
9) External connections
This covers issues like those the Eclipse team is experiencing.
If, for example, a TCP connection to the seed is severed, it causes
Open RTE to call abort, which means Eclipse just aborted. That's
not so good. There are other naming / status issues that also need
to be handled here.
Assigned to:
9.5) Fix/Complete orte-ps and friends
orte-ps / orte-clean / etc. all depend on being able to make a
connection to the orte universe that doesn't result in bad things
happening. We should finish these things for obvious reasons.
Assigned to:
10) Remote connections
This is similar to #9, but includes the ability to start a remote
HNP process
Assigned to:
11) Dynamic MPI-2 support
ORTE's support for the MPI-2 dynamics has some well-known issues.
In addition, we need to change some behaviors at the Open MPI level
to behave better.
Assigned to:
12) XCPU support
The XCPU system is a distributed process management system
implemented using the Plan 9 filesystem. An RAS (possibly) and PLS
are needed to support launching on XCPU systems.
Assigned to:
13) Multi-cell support
Assigned to:
14) Memory usage / null components
This is related to an e-mail Ralph or Jeff sent yesterday regarding
support for NULL components. The idea is to not load all the
components into memory if null is specified as the prefered
component name.
Assigned to:
15) RAS multi-component issues
If you are in an allocation (say, TM or BProc) and try to specify
--hostfile on the orterun command line, the hostfile option will be
ignored and you'll use the previous allocation. There are some
other similar cases, all of which can result in rather unexpected
behaviour from the user's point of view
Assigned to:
|
![[Flag]](../../../../../community/mirrors/flag-us.gif){kind=small}