Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: Re: [OMPI users] MPI_Alltoallv and unknown data send sizes
From: Daniel Spångberg (daniels_at_[hidden])
Date: 2008-09-10 12:30:29

George, thanks for the quick answer!

I thought about using alltoall before the alltoallv, but it "feels" like
this might end up slow having two alltoall, at least doubling the latency.
Might still be faster than a large bunch of sendrecvs of course. I'll
simply have to do some short tests, anyway if it turns out the
alltoall/alltoallv combo is too slow.

Thanks again!

Den 2008-09-10 17:10:06 skrev George Bosilca <bosilca_at_[hidden]>:

> Daniel,
> Your understanding of he MPI standard requirement with regard to
> MPI_Alltoallv is now 100% accurate. The send count and datatype should
> match what the receiver expect. You can always use an MPI_Alltoall
> before the MPI_Alltoallv to exchange the lengths that you expect.
> george.
> On Sep 10, 2008, at 1:46 PM, Daniel Spångberg wrote:
>> Dear all,
>> First some background, the real question is at the end of this
>> (longish) mail.
>> I have a problem where I need to exchange data between all processes.
>> The data is unevenly distributed and I thought at first I could use
>> MPI_Alltoallv to transfer the data. However, in my case, the receivers
>> do not know how many data items the senders will send, but it is
>> relatively easy to set up so the receiver can figure out the maximum
>> number of items the sender will send, so I set the recvcounts to the
>> maximum possible, and the sendcounts to the actual number of elements
>> (smaller than recvcounts).
>> The mpi-forum description (from
>> describes the
>> following:
>> MPI_ALLTOALLV(sendbuf, sendcounts, sdispls, sendtype, recvbuf,
>> recvcounts, rdispls, recvtype, comm)
>> IN sendbuf starting address of send buffer (choice)
>> IN sendcounts integer array equal to the group size specifying the
>> number of elements to send to each processor
>> IN sdispls integer array (of length group size). Entry j specifies the
>> displacement (relative to sendbuf) from which to take the outgoing data
>> destined for process j
>> IN sendtype data type of send buffer elements (handle)
>> OUT recvbuf address of receive buffer (choice)
>> IN recvcounts integer array equal to the group size specifying the
>> number of elements that can be received from each processor
>> IN rdispls integer array (of length group size). Entry i specifies the
>> displacement (relative to recvbuf) at which to place the incoming data
>> from process i
>> IN recvtype data type of receive buffer elements (handle)
>> IN comm communicator (handle)
>> In particular the wording is "the number of elements that can be
>> received from each processor" for recvcounts, and does not say that
>> this must be exactly the same as the number of elements sent.
>> It also mentions that it should work similarly as a number of
>> independent MPI_Send/MPI_Recv calls. The amount of data sent in such a
>> case does not need to exactly match the amount of data received.
>> I, unfortunately, missed the following:
>> The type signature associated with sendcounts[j], sendtypes[j] at
>> process i must be equal to the type signature associated with
>> recvcounts[i], recvtypes[i] at process j. This implies that the amount
>> of data sent must be equal to the amount of data received, pairwise
>> between every pair of processes. Distinct type maps between sender and
>> receiver are still allowed.
>> And the openmpi man page shows
>> When a pair of processes exchanges data, each may pass
>> different ele-
>> ment count and datatype arguments so long as the sender
>> specifies the
>> same amount of data to send (in bytes) as the receiver
>> expects to
>> receive.
>> I did test my program on different send/recv counts, and while it
>> sometimes works, sometimes it does not. Even if it worked I would not
>> be comfortable using it anyway.
>> The question is: If there is no way of determining the length of the
>> data sent by the sender on the receiving end, I see two options: Either
>> always transmit too much data using MPI_Alltoall(v) or cook up my own
>> routine based on PTP calls, probably MPI_Sendrecv is the best option.
>> Am I missing something?
>> --Daniel Spångberg
>> Materials Chemistry
>> Uppsala University
>> Sweden
>> _______________________________________________
>> users mailing list
>> users_at_[hidden]
> _______________________________________________
> users mailing list
> users_at_[hidden]

Daniel Spångberg
Uppsala Universitet