Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: Re: [OMPI users] openMPI, transfer data from multiple sources to one destination
From: Eugene Loh (Eugene.Loh_at_[hidden])
Date: 2009-01-05 12:32:47


Jack Bryan wrote:
I need to transfer data from multiple sources to one destination.
The requirement is:

(1) The sources and destination nodes may work asynchronously.
 
(2) Each source node generates data package in their own paces.
    And, there may be many packages to send. Whenever, a data package
    is generated , it should be sent to the desination node at once.
    And then, the source node continue to work on generating the next
    package.

(3) There is only one destination node , which must receive all data
    package generated from the source nodes.
    Because the source and destination nodes may work asynchronously,
    the destination node should not wait for a specific source node until
    the source node sends out its data.

    The destination node should be able to receive data package
    from anyone source node whenever the data package is available in a
    source node.

My question is :

What MPI function should be used to implement the protocol above ?

If I use MPI_Send/Recv, they are blocking function. The destination
node have to wait for one node until its data is available.

The communication overhead is too high.

If I use MPI_Bsend, the destination node has to use MPI_Recv to ,
a Blocking receive for a message .

This can make the destination node wait for only one source node and
actually other source nodes may have data avaiable.
You've already gotten a number of suggestions, but I thought I'd add my $0.02 to help you (hopefully) sort them out.

I agree with Terry that you should read up on MPI_ANY_SOURCE.  With this, you can listen for any source process and use functions like MPI_Recv or MPI_Irecv.  I further agree that you could consider functions like MPI_Probe or MPI_Iprobe.

Win's examples don't seem to apply since they implement gathers (one input from each process) and you explicitly said that no such regularity can be assumed.

Biagio's suggestion might be helpful, but also sounds like it might be overkill.