Open MPI logo

Open MPI Development Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Development mailing list

From: Brian Barrett (brbarret_at_[hidden])
Date: 2006-08-24 16:16:26


Hi all -

LANL had an internal meeting yesterday trying to classify a number of
issues we're having with the run-time environment for Open MPI and how
to best prioritize team resources. We thought it would be good to
both share the list (with priorities) with the group and to ask the
group if there were other issues that need to be addressed (either
short or long term). We've categorized the issues as performance
related, robustness, and feature / platform support. The numbers are
the current priority on our list, and items within a category are
sorted by priority.

PERFORMANCE:

5) 50% scale factor in process startup

   Start-up of non-MPI jobs has a strange bend in the timing curve
   when the number of processes we are trying to start is greater than
   or equal to 50% of the current allocation. It appears that
   starting a 16 process (1 ppn) job takes longer if there are 32
   nodes in the allocation than if there are 64 nodes in the
   allocation.

   Assigned to: Galen

6) MPI_INIT startup timings

   In addition to seeming to suffer from the same 50% issue as the
   previous issue, there also appears to be a number of places in
   MPI_INIT where we spend a considerable amount of time when at
   scale, leading to startup times much worse than LA-MPI or
   MPIEXEC/MVAPICH.

   Assigned to: Galen

ROBUSTNESS:

1) MPI process aborting issue

   This is the orted spin, MPI processes don't die, etc. issue that
   occurs when some process dies unexpectedly. Ralph has already sent
   a detailed e-mail to devel about this issue.

   Assigned to: Ralph

1.5) MPI_ABORT rework

   The MPI process aborting issue is going to require a rework of
   MPI_ABORT so that it uses the error manager instead of calling
   terminate_proc/terminate_job.

   Assigned to: Brian

2) ORTE hangs when start-up fails

   If an orted fails to start or fails to connect back to the HNP, the
   system hangs waiting for the callback. If a orted process fails to
   start entirely, we sometimes catch this. But we need a better
   mechanism for handling the general failure case.

   Assigned to: Ralph

3) Hardened cleanup of session directory

   While #1 should greatly help in ensuring that the session directory
   is cleaned up every time, there are still a number of race
   conditions that need to be sorted out. The goal is to develop a
   plan that ensures files that need to be removed are removed
   automatically a high percentage of the time, that there is a way to
   allow a tool like orte_clean to clean up everything it should clean
   up, and that there is a way to make sure files that should not be
   automatically removed aren't automatically removed.

   Assigned to: Brian

3.5) Process not found hangs

   See https://svn.open-mpi.org/trac/ompi/ticket/245

   Assigned to: Ralph

7) Node death failures / hangs

   With the exception of BProc, if a node fails, we don't detect the
   failure. Even if we did detect the failure, we have no general
   mechanism for dealing with that failure. The bulk of this project
   is going to be adding a general SOH/SMR component that uses the OOB
   for timeout pings.

   Assigned to: Brian

15) More friendly error messages

   There are situations where we give something south of a useful
   error message when an error is found. We should play nicer with
   users.

   Assigned to:

16) Consistent error checking

   We've had a number of recent instances of errors occuring, but not
   being propogated / returned to the user simply because no one ever
   checked the return code. We need to audit most of ORTE to always
   check return codes.

   Assigned to:

FEATURE / PLATFORM SUPPORT:

4) TM error handling

   TM, while used on a number of large systems LANL needs to support,
   is not exactly friendly to usage at scale. It seems that it likes
   to go away and cry to mamma for a couple seconds, returning system
   error messages, only to come back and be ok a second later. This
   means that every TM call needs to be handled as if it's going to
   fail, and we need to be prepared to re-initialize the system (if
   possible) when failures occur. In testing on t-bird, launching was
   usually pretty stable, but the calls to get the node allocations
   tended to result in the strange behavior. These should definitely
   be re-startable type errors

   Assigned to: Brian

8) Hetergeneous Issues

   Assigned to:

9) External connections

   This covers issues like those the Eclipse team is experiencing.
   If, for example, a TCP connection to the seed is severed, it causes
   Open RTE to call abort, which means Eclipse just aborted. That's
   not so good. There are other naming / status issues that also need
   to be handled here.

   Assigned to:

9.5) Fix/Complete orte-ps and friends

    orte-ps / orte-clean / etc. all depend on being able to make a
    connection to the orte universe that doesn't result in bad things
    happening. We should finish these things for obvious reasons.

    Assigned to:

10) Remote connections

   This is similar to #9, but includes the ability to start a remote
   HNP process

   Assigned to:

11) Dynamic MPI-2 support

   ORTE's support for the MPI-2 dynamics has some well-known issues.
   In addition, we need to change some behaviors at the Open MPI level
   to behave better.

   Assigned to:

12) XCPU support

   The XCPU system is a distributed process management system
   implemented using the Plan 9 filesystem. An RAS (possibly) and PLS
   are needed to support launching on XCPU systems.

   Assigned to:

13) Multi-cell support

   Assigned to:

14) Memory usage / null components

   This is related to an e-mail Ralph or Jeff sent yesterday regarding
   support for NULL components. The idea is to not load all the
   components into memory if null is specified as the prefered
   component name.

   Assigned to:

15) RAS multi-component issues

   If you are in an allocation (say, TM or BProc) and try to specify
   --hostfile on the orterun command line, the hostfile option will be
   ignored and you'll use the previous allocation. There are some
   other similar cases, all of which can result in rather unexpected
   behaviour from the user's point of view

   Assigned to: