Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: [OMPI users] How to reduce Isend & Irecv bandwidth?
From: Thomas Watson (exascale.system_at_[hidden])
Date: 2013-05-01 13:28:26


Hi,

I have a program where each MPI rank hosts a set of data blocks. After
doing computation over *some of* its local data blocks, each MPI rank needs
to exchange data with other ranks. Note that the computation may involve
only a subset of the data blocks on a MPI rank. The data exchange is
achieved at each MPI rank through Isend and Irecv and then Waitall to
complete the requests. Each pair of Isend and Irecv exchanges a
corresponding pair of data blocks at different ranks. Right now, we do
Isend/Irecv for EVERY block!

The idea is that because the computation at a rank may only involves a
subset of blocks, we could mark those blocks as dirty during the
computation. And to reduce data exchange bandwidth, we could only exchanges
those *dirty* pairs across ranks.

The problem is: if a rank does not compute on a block 'm', and if it does
not call Isend for 'm', then the receiving rank must somehow know this and
either a) does not call Irecv for 'm' as well, or b) let Irecv for 'm' fail
gracefully.

My questions are:
1. how Irecv will behave (actually how MPI_Waitall will behave) if the
corresponding Isend is missing?

2. If we still post Isend for 'm', but because we really do not need to
send any data for 'm', can I just set a "flag" in Isend so that MPI_Waitall
on the receiving side will "cancel" the corresponding Irecv immediately?
For example, I can set the count in Isend to 0, and on the receiving side,
when MPI_Waitall see a message with empty payload, it reclaims the
corresponding Irecv? In my code, the correspondence between a pair of Isend
and Irecv is established by a matching TAG.

Thanks!

Jacky