Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: Re: [OMPI users] Deadlock question
From: Gijsbert Wiesenekker (gijsbert.wiesenekker_at_[hidden])
Date: 2010-05-27 01:23:41


On May 24, 2010, at 20:27 , Eugene Loh wrote:

> Gijsbert Wiesenekker wrote:
>
>> My MPI program consists of a number of processes that send 0 or more messages (using MPI_Isend) to 0 or more other processes. The processes check periodically if messages are available to be processed. It was running fine until I increased the message size, and I got deadlock problems. Googling learned I was running into a classic deadlock problem if (see for example http://www.cs.ucsb.edu/~hnielsen/cs140/mpi-deadlocks.html). The workarounds suggested like changing the order of MPI_Send and MPI_Recv do not work in my case, as it could be that one processor does not send any message at all to the other processes, so MPI_Recv would wait indefinitely.
>> Any suggestions on how to avoid deadlock in this case?
>>
> The problems you describe would seem to arise with blocking functions like MPI_Send and MPI_Recv. With the non-blocking variants MPI_Isend/MPI_Irecv, there shouldn't be this problem. There should be no requirement of ordering the functions in the way that web page describes... that workaround is suggested for the blocking calls. It feels to me that something is missing from your description.
>
> If you know the maximum size any message will be, you can post an MPI_Irecv with wild card tags and source ranks. You can post MPI_Isend calls for whatever messages you want to send. You can use MPI_Test to check if any message has been received; if so, process the received message and re-post the MPI_Irecv. You can use MPI_Test to check if any send messages have completed; if so, you can reuse those send buffers. You need some signal to indicate to processes that no further messages will be arriving.
> _______________________________________________
> users mailing list
> users_at_[hidden]
> http://www.open-mpi.org/mailman/listinfo.cgi/users

My program was running fine using the methods you describe (MPI_Isend/MPI_Test/MPI_Irecv), until I increased the message size. My program was not running very efficient because of the MPI overhead associated with sending/receiving a large number of small messages. So I decided to combine messages before sending them, and then I got the deadlock problems: the MPI_Test calls never returned true, so the MPI_Isend calls never completed. As described on the link given above, the reason was that I exhausted the MPI system buffer space, in combination with the unsafe ordering of the send/receive calls (but I cannot see how I can change that order given the nature of my program).
See for example also http://publib.boulder.ibm.com/infocenter/clresctr/vxrx/index.jsp?topic=/com.ibm.cluster.pe.doc/pe_422/am10600481.html: 'Destination buffer space unavailability cannot cause a safe MPI program to fail, but could cause hangs in unsafe MPI programs. An unsafe program is one that assumes MPI can guarantee system buffering of sent data until the receive is posted.'

Gijsbert