Tee Wen Kai -
You asked "Just to find out more about the consequences for exiting MPI processes without calling MPI_Finalize, will it cause memory leak or other fatal problem?"
Be aware that Jeff has offered you an OpenMPI implementation oriented answer rather than an MPI standard oriented answer.
When there is a communicator involving 2 or more tasks and any task involved in that communicator goes down, all other tasks that are members of that communicator enter a state the MPI standard says cannot be trusted. It is legitimate for the process that manages an MPI job as a single entity to recognize that the loss of a member task has made the state of all connected tasks untrustworthy and bring down all previously connected tasks too.
When you use MPI_Comm_spawn, one result is an intercommunicator connecting the task that did the spawn to the task(s) that were spawned so the two sides are "connected". If you intend to use MPI to communicate between the spawn caller and the spawned tasks they must remain connected. You can explicitly disconnect them and then a failure of the spawned task is harmless to the task that spawned it but doing the disconnect costs you the communication path.
The MPI standard does not require that connected tasks be brought down but it is a valid MPI implementation behavior. This makes some sense when you consider the fact that there is no MPI mechanism by which the other tasks can see that the communicator involving the lost task is now broken and there is no way a collective communication can work "correctly" on a communicator that has lost a member task.
For example, what would it mean to call MPI_Reduce on MPI_COMM_WORLD when a member of MPI_COMM_WORLD has been lost (especially if it is the root that was lost)? If you had an MPI application that computed for hours between the loss of one task and the next collective call on MPI_ COMM_WORLD, would you prefer to pay for hours of computation and then deadlock at the collective call or just abort ASAP after the job is recognizably broken.
There is a fault tolerance working group trying to define something for MPI 3.0 but at this stage they are still trying to work out a proposal to bring before the MPI Forum. You might be interested in getting involved in that effort. They try to address question like:
- how would a task know it should not make collective calls on the broken communicator?
- should the communicator still support point to point communications with remaining tasks?
- If a task has posted a receive and the expected sender is then lost, how should the posted receive act?
- is there a clean way to "repair" the broken communicator by spawning a replacement task?
- is there a clean way to "shrink" the broken communicator
The Fault Tolerance Working Group has taken on a very tough problem. The list above is just a tiny sample of the challenges in making MPI fault tolerant.
Dick Treumann - MPI Team
IBM Systems & Technology Group
Dept X2ZA / MS P963 -- 2455 South Road -- Poughkeepsie, NY 12601
Tele (845) 433-7846 Fax (845) 433-8363
<graycol.gif>Jeff Squyres ---06/04/2009 07:32:25 AM---On Jun 4, 2009, at 2:16 AM, Tee Wen Kai wrote: > Just to find out more about the consequences for ex
Jeff Squyres <email@example.com>
"Open MPI Users" <firstname.lastname@example.org>
06/04/2009 07:32 AM
Re: [OMPI users] Exit Program Without Calling MPI_Finalize ForSpecial Case
On Jun 4, 2009, at 2:16 AM, Tee Wen Kai wrote:
> Just to find out more about the consequences for exiting MPI
> processes without calling MPI_Finalize, will it cause memory leak or
> other fatal problem?
If you're exiting the process, you won't cause any kind of problems --
the OS will clean up everything.
However, we might also have the orted clean up some things when MPI
processes unexpectedly die (e.g., filesystem temporary files in /
tmp). So you might want to leave those around to clean themselves up
and die naturally.
users mailing list
users mailing list