Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: [OMPI users] Troubles using MPI_Isend/MPI_Irecv/MPI_Waitany and MPI_Allreduce
From: Pedro Gonnet (gonnet_at_[hidden])
Date: 2011-10-20 05:42:37

Hi all,

I am currently working on a multi-threaded hybrid parallel simulation
which uses both pthreads and OpenMPI. The simulation uses several
pthreads per MPI node.

My code uses the nonblocking routines MPI_Isend/MPI_Irecv/MPI_Waitany
quite successfully to implement the node-to-node communication. When I
try to interleave other computations during this communication, however,
bad things happen.

I have two MPI nodes with two threads each: one thread (i) doing the
nonblocking communication and the other (ii) doing other computations.
At some point, the threads (ii) need to exchange data using
MPI_Allreduce, which fails if the first thread (i) has not completed all
the communication, i.e. if thread (i) is still in MPI_Waitany.

Using the in-place MPI_Allreduce, I get a re-run of this bug: If I
don't use in-place, the call to MPI_Waitany (thread ii) on one of the
MPI nodes waits forever.

My guess is that when the thread (ii) calls MPI_Allreduce, it gets
whatever the other node sent with MPI_Isend to thread (i), drops
whatever it should have been getting from the other node's
MPI_Allreduce, and the call to MPI_Waitall hangs.

Is this a known issue? Is MPI_Allreduce not designed to work alongside
the nonblocking routines? Is there a "safe" variant of MPI_Allreduce I
should be using instead?

I am using OpenMPI version 1.4.3 (version 1.4.3-1ubuntu3 of the package
openmpi-bin in Ubuntu). Both MPI nodes are run on the same dual-core
computer (Lenovo x201 laptop).

If you need more information, please do let me know! I'll also try to
cook-up a small program reproducing this problem...

Cheers and kind regards,