Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: Re: [OMPI users] MPI_Recv hangs
From: Jorge Chiva Segura (jordic_at_[hidden])
Date: 2012-05-09 10:19:46


On Wed, 2012-05-09 at 15:24 +0200, Eduardo Morras wrote:
> Sorry for the delay, and sorry again because in last mail i had the
> wrong taste that it was some kind of homework problem.
Don't worry ;).
I simplified the core of the problem just to make it easier to
understand (at least that was my intention xD) . And I wrote all the
information that I found relevant in the opening post (320 CPU's,
versions of OMPI, operative system and infiniband, etc.) precisely
because I wanted to show that it wasn't a homework or anything like
that.
>
> At 17:41 04/05/2012, you wrote:
> > > The logic of send/recv looks ok. Now, in 5 and 7, recvSize(p2) and
> > > recvSize(p1) function what value returns?
> >All the sendSizes and RecvSizes are constant between iterations and are
> >calculated as a setup before all the calculations start.
>
> <snip>
>
> >Do you know what could cause the program to hang with the default value
> >(310) and to work fine with 305? I also tested it with 311 but it hanged
> >so it seems that it is not enough to activate the SEND flag.
>
> On your code, the only point where it could fail is if one of the
> precalculated message size values is wrongly calculated and executes
> the Recieve where it shouldn't.
Yes, but after the sizes are calculated they don't change and that's why
I find it weird to hang the 30th time the whole communication loop is
executed :S .
>
> From previous mails i understand that no if(ok!=MPI... line fires
> and there's no Sender waiting. The Ssend ends when the Recv starts to
> receive, not when the Recv ends the receive, so the sender may get an
> Ok but if there's an error Recv keeps the block. As you are using
> blocking communications, you can't do anything to prevent this, for
> example, check the Recv status while receiving.
I don't know how to check the Recv status because the processor remains
waiting for the message at the Recv function.
> Try to use Send instead Ssend (it should work but it could hang too)
> or change design to a non-blocking approach.

The problem is that it also hangs with non-blocking communications. The
real program is coded with non-blocking communications and it started to
hang when the size of the mesh got bigger. I just changed to blocking
communications to easy the debugging task.

Now it works, with blocking and non-blocking communications, just
changing the value of the mca parameter btl_openib_flags to 304 or 305
(the default value is 310). That means that the problem is with the RDMA
protocols in infiniband for large messages. As far as I know, with those
values the flags GET(4) and PUT(2) are deactivated and the protocol for
large messages remains the same as the one for small messages
(send/receive). For me, it seems that there is a bug (problably a memory
leak) in OMPI or OFED.

Thanks for your help,
Jorge

-- 
Aquest missatge ha estat analitzat per MailScanner
a la cerca de virus i d'altres continguts perillosos,
i es considera que està net.