Open MPI logo

Open MPI Development Mailing List Archives

  |   Home   |   Support   |   FAQ   |  

This web mail archive is frozen.

This page is part of a frozen web archive of this mailing list.

You can still navigate around this archive, but know that no new mails have been added to it since July of 2016.

Click here to be taken to the new web archives of this list; it includes all the mails that are in this frozen archive plus all new mails that have been sent to the list since it was migrated to the new archives.

Subject: Re: [OMPI devel] RFC: Fix missing code in MPI_Abort functionality
From: George Bosilca (bosilca_at_[hidden])
Date: 2011-06-09 18:13:32


What I'm saying is that there is no reason to have any other type of MPI_Abort if we are not able to compute the set of connected processes.

With this RFC the processes on the communicator on MPI_Abort will abort. Then the other processes in the same MPI_COMM_WORLD (in fact jobid) will be notified (if we suppose that the ORTE will not make a difference between aborted and faulty). As a result the entire MPI_COMM_WORLD will be aborted, if we consider a sane application where everyone use the same type of error handler. However, this is not enough. We have to distribute the abort signal to every other process "connected", and I don't see how we can compute this list of connected processes in Open MPI today.It is not that I don't see it in your patch, it is that the definition of the connectivity in the MPI standard is transitive and relies heavily on a correct implementation for the MPI_Comm_disconnect.

  george.

On Jun 9, 2011, at 16:59 , Josh Hursey wrote:

> On Thu, Jun 9, 2011 at 4:47 PM, George Bosilca <bosilca_at_[hidden]> wrote:
>> If this change the behavior of MPI_Abort to only abort processes on the specified communicator how this doesn't affects the default user experience (when today it aborts everything)?
>
> Open MPI does abort everything by default - decided by the runtime at
> the moment (but addressed in your RFC). So it does not matter if one
> process aborts or if many do. So the behavior of MPI_Abort experienced
> by the user will not change. Effectively the only change is an extra
> message in the runtime before the process actually calls
> errmgr.abort().
>
> This branch just makes the implementation complete by first telling
> ORTE that a group of processes, defined by the communicator, should be
> terminated along with the calling process. Currently ORTE notices that
> there was an abort, and terminates the job. Once your RFC goes through
> then this may no longer be the case, and OMPI can determine what to do
> when it receives a process failure notification.
>
>>
>> If we accept the fact that MPI_Abort will only abort the processes in the current communicator what happens with the other processes in the same MPI_COMM_WORLD (but not on the communicator that has been used by MPI_Abort)?
>
> Currently, ORTE will abort them as well. When your RFC goes through
> then the OMPI layer will be notified of the error and can take the
> appropriate action, as determined by the MPI standard.
>
>> What about all the other connected processes (based on the connectivity as defined in the MPI standard in Section 10.5.4) ? Do they see this as a fault?
>
> They are informed of the fault via the ORTE errmgr callback routine
> (that we have an RFC for), and then can take the appropriate action
> based on MPI semantics. So we are pushing the decision of the
> implication of the fault to the OMPI layer - where it should be.
>
>
> The remainder of the OMPI layer logic for MPI_ERRORS_RETURN and other
> connected error management scenarios is not included in this patch
> since that depends on there being a callback to the OMPI layer - which
> does not exist just yet. So a small patch to wire in the ORTE piece to
> allow the OMPI layer to request a set of processes to be terminated -
> to more accurately support MPI_Abort semantics.
>
> Does that answer your questions?
>
> -- Josh
>
>
>>
>> george.
>>
>> On Jun 9, 2011, at 16:32 , Josh Hursey wrote:
>>
>>> WHAT: Fix missing code in MPI_Abort
>>>
>>> WHY: MPI_Abort is missing logic to ask for termination of the process
>>> group defined by the communicator
>>>
>>> WHERE: Mostly orte/mca/errmgr
>>>
>>> WHEN: Open MPI trunk
>>>
>>> TIMEOUT: Tuesday, June 14, 2011 (after teleconf)
>>>
>>> Details:
>>> -------------------------------------------
>>> A bitbucket branch is available here (last sync to r24757 of trunk)
>>> https://bitbucket.org/jjhursey/ompi-abort/
>>>
>>> In the MPI Standard (v2.2) Section 8.7 after the introduction of
>>> MPI_Abort, it states:
>>> "This routine makes a best attempt to abort all tasks in the group of comm."
>>>
>>> Open MPI currently only calls orte_errmgr.abort() to abort the calling
>>> process itself. The code to ask for the abort of the other processes
>>> in the group defined by the communicator is commented out. Since one
>>> process calling abort currently causes all processes in the job to
>>> abort, it has not been a big deal. However as the group starts
>>> exploring better resilience in the OMPI layer (with further support
>>> from the ORTE layer) this aspect of MPI_Abort will become more
>>> necessary to get right.
>>>
>>> This branch adds back the logic necessary for a single process calling
>>> MPI_Abort to request, from ORTE errmgr, that a defined subgroup of
>>> processes be aborted. Once the request is sent to the HNP, the local
>>> process then calls abort on itself. The HNP requests that the defined
>>> subgroup of processes be terminated using the existing plm mechanisms
>>> for doing so.
>>>
>>> This change has no effect on the current default user experienced
>>> behavior of MPI_Abort.
>>>
>>> --
>>> Joshua Hursey
>>> Postdoctoral Research Associate
>>> Oak Ridge National Laboratory
>>> http://users.nccs.gov/~jjhursey
>>> _______________________________________________
>>> devel mailing list
>>> devel_at_[hidden]
>>> http://www.open-mpi.org/mailman/listinfo.cgi/devel
>>
>>
>> _______________________________________________
>> devel mailing list
>> devel_at_[hidden]
>> http://www.open-mpi.org/mailman/listinfo.cgi/devel
>>
>>
>
>
>
> --
> Joshua Hursey
> Postdoctoral Research Associate
> Oak Ridge National Laboratory
> http://users.nccs.gov/~jjhursey
>
> _______________________________________________
> devel mailing list
> devel_at_[hidden]
> http://www.open-mpi.org/mailman/listinfo.cgi/devel