Open MPI logo

Open MPI Development Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Development mailing list

Subject: Re: [OMPI devel] hcoll destruction via MPI attribute
From: Joshua Ladd (joshual_at_[hidden])
Date: 2014-01-09 12:05:31

See inline...

-----Original Message-----
From: devel [mailto:devel-bounces_at_[hidden]] On Behalf Of Jeff Squyres (jsquyres)
Sent: Thursday, January 09, 2014 11:53 AM
To: Open MPI Developers
Cc: Devendar Bureddy; valentin (valentin.petrov_at_[hidden]) (valentin.petrov_at_[hidden]); Mike Dubman
Subject: Re: [OMPI devel] hcoll destruction via MPI attribute

On Jan 9, 2014, at 11:00 AM, Joshua Ladd <joshual_at_[hidden]> wrote:

> Hcoll uses the PML as an "OOB" to bootstrap itself. When a communicator is destroyed, by the time we destroy the hcoll module, the communicator context is no longer valid and any pending operations that rely on its existence will fail. In particular, we have a non-blocking synchronization barrier that may be in progress when the communicator is destroyed.

Can you explain this a little more? Do you mean you have a pending MPI_Ibarrier running on that communicator? (i.e., the ibarrier has started but not completed) Or you have some started-but-not-completed MPI_Isends/MPI_Irecvs?

(using the PML/coll equivalents of these of course -- not the top-level MPI_* foo functions)

Or are you saying that you need the destruction of the hcoll module on a given communicator to be synchronous between all processes in that communicator?

[Josh] We have a recursive doubling algorithm in progress implemented with PML send/recvs, more accurately , with "RTE_isend/RTE_irecv" functions, which, in the case of OMPI are PML calls.

> Registering the delete callback allows us to finish these operations because the context is still valid inside of this callback. The commented out code is the "prototype" protocol that attempted to handle this scenario in an entirely different (and more complex) way. It is not needed now. We don't want to introduce solutions that are OMPI specific, because we need to be able to integrate hcoll into other runtimes. We considered approaching the community about changing the comm destroy flow in OMPI to keep the context alive long enough to complete our synchronization barriers, but then the solution is tied to a particular MPI

I'm not quite sure I understand -- the hcoll module (where this code is located) is completely OMPI-specific. I thought that libhcoll was your independent-of-MPI-implementations portion of this code...?

[Josh] The hcoll module is the integration layer. HCOLL is completely standalone. When you create a new communicator, you create a new hcoll module whih in turn creates a new "hcoll context". We have defined what we call the RTE interface, which is an API that runtimes need to implement in order to use HCOLL. Basically, the runtime needs to provide HCOLL with a handle to a non-blocking send and receive, implement some callbacks, and pass in a group handle. HCOLL is completely MPI agnostic.

Jeff Squyres
For corporate legal information go to:
devel mailing list