Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: Re: [OMPI users] How to quit asynchronous processes
From: Richard Treumann (treumann_at_[hidden])
Date: 2009-02-23 14:20:42


I am not 100% sure I understand your situation. It it this?

Process A has an ongoing stream of inputs. For each input unit, A does some
processing and then passes on work to B via a message. B receives the
message from A and does some additional work before sending a message to C.
C receives the message from B and does more work before spitting out an
answer. At any time, whoever is interested in the answers coming from C
can tell C to quit spitting out answers and shut down the pipeline.

Is this right?

If so, I think the best answer is to let the shutdown flow through the same
pipeline, flushing the messages ahead of the shutdown. The reason for this
is that an MPI program cannot be cleanly terminated until all sends are
matched by receives. If A has tried to send a message to B but B has not
yet received it and you try to terminate B, you guarantee A cannot
terminate cleanly.

If the message A sends B and the message B sends C can be designed to
indicate "shutdown" then here is the clean solution. For example, if you
know the first word of a working message is always a positive number, you
could send a message that has a -1 in the first word to indicated
"shutdown". As each message arrives at B or C, the first step is to check
for -1.

When C is told to shut down, it sends a message to A that says "flush the
pipe". Before each time A picks up a new input to process, A does an
Iprobe for a "flush the pipe" message from C . If C has sent a "flush the
pipe" message, A receives that message, A discards the input, sends a
"shutdown" to B and calls MPI_Finalize. Each time B gets a message from A
it looks to see if the message is a "shutdown" or real work. If it is a
"shutdown", B forwards the "shutdown" to C and B calls MPI_Finalize. Each
time C gets a message from B it looks to see if the message is a "shutdown"
or real work. If it is a "shutdown", then C calls MPI_Finalize.

The pipeline has been flushed so none of them has an unmatched send or
receive. A and B are waiting in MPI_Finalize by the time C actually calls
MPI_Finalize so everything cleans up neatly.

Dick Treumann - MPI Team
IBM Systems & Technology Group
Dept X2ZA / MS P963 -- 2455 South Road -- Poughkeepsie, NY 12601
Tele (845) 433-7846 Fax (845) 433-8363

users-bounces_at_[hidden] wrote on 02/22/2009 10:49:27 PM:

> [image removed]
>
> [OMPI users] How to quit asynchronous processes:
>
> Tee Wen Kai
>
> to:
>
> users
>
> 02/22/2009 10:50 PM
>
> Sent by:
>
> users-bounces_at_[hidden]
>
> Please respond to teewenkai, Open MPI Users
>
> Hi,
>
> I have three processes A,B & C. A will receive some data and process
> and after that will pass results to B. Thereafter, A will go back to
> get data and process and prepare to send them to B again. Same goes
> for B to C where C will provide the final output. As all the
> processes are running asynchronously, I can't find a method to quit
> all processes gracefully. I have tried to use C to issue MPI_Abort
> when quit command is received with processes A and B entering the
> SIGTERM handler to free all the allocated memories. However, the
> processes may be in the midst of a communication when SIGTERM signal
> is raise, resulting in some error msg like "remote connection close"
> etc. To prevent such error messages from displaying, I couldn't use
> MPI_Finalize.
>
> I seek your advice on methods to quit asynchonous processes properly
> without all these error messages. And I would also like to find out
> what will be the outcome eventually if I don't close the processes
> with an MPI_Finalize command, will it lead to eventual memory leak etc?
>
> Thank you very much.
>
> Regards
> Wenkai
>
>
> Get your preferred Email name!
> Now you can @ymail.com and @rocketmail.com.
> _______________________________________________
> users mailing list
> users_at_[hidden]
> http://www.open-mpi.org/mailman/listinfo.cgi/users