Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: Re: [OMPI users] Coding help requested
From: Eugene Loh (Eugene.Loh_at_[hidden])
Date: 2009-11-10 15:45:48


amjad ali wrote:

> Hi all.
> (sorry for duplication, if it is)
>
> I have to parallelize a CFD code using domain/grid/mesh partitioning
> among the processes. Before running, we do not know,
> (i) How many processes we will use ( np is unknown)
> (ii) A process will have how many neighbouring processes (my_nbrs = ?)
> (iii) How many entries a process need to send to a particular
> neighbouring process.
> But when the code run, I calculate all of this info easily.
>
>
> The problem is to copy a number of entries to an array then send that
> array to a destination process. The same sender has to repeat this
> work to send data to all of its neighbouring processes. Is this
> following code fine:
>
> DO i = 1, my_nbrs
> DO j = 1, few_entries_for_this_neighbour
> send_array(j) = my_array(jth_particular_entry)
> ENDDO
> CALL MPI_ISEND(send_array(1:j),j, MPI_REAL8, dest(i), tag,
> MPI_COMM_WORLD, request1(i), ierr)

instead of "j" I assume you intended something like
"few_entries_for_this_neighbour"

> ENDDO
>
> And the corresponding receives, at each process:
>
> DO i = 1, my_nbrs
> k = few_entries_from_this_neighbour
> CALL MPI_IRECV(recv_array(1:k),k, MPI_REAL8, source(i), tag,
> MPI_COMM_WORLD, request2(i), ierr)
> DO j = 1, few_from_source(i)
> received_data(j) = recv_array(j)
> ENDDO
> ENDDO
>
> After the above MPI_WAITALL.
>
>
> I think this code will not work. Both for sending and receiving. For
> the non-blocking sends we cannot use send_array to send data to other
> processes like above (as we are not sure for the availability of
> application buffer for reuse). Am I right?
>
> Similar problem is with recv array; data from multiple processes
> cannot be received in the same array like above. Am I right?

Correct for both send and receive. When you call MPI_Isend, the buffer
cannot be written until the MPI_Waitall. When you use MPI_Irecv, you
cannot read the data until MPI_Waitall. You're reusing both send and
receive buffers too often and too soon.

> Target is to hide communication behind computation. So need non
> blocking communication. As we do know value of np or values of my_nbrs
> for each process, we cannot decide to create so many arrays. Please
> suggest solution.

You can allocate memory dynamically, even in Fortran.

> A more subtle solution that I could assume is following:
>
> cc = 0
> DO i = 1, my_nbrs
> DO j = 1, few_entries_for_this_neighbour
> send_array(cc+j) = my_array(jth_particular_entry)
> ENDDO
> CALL MPI_ISEND(send_array(cc:cc+j),j, MPI_REAL8, dest(i), tag,
> MPI_COMM_WORLD, request1(i), ierr)
> cc = cc + j
> ENDDO

Same issue with j as before, but yes concatenating the various send
buffers in a one-dimensional fashion should work.

> And the corresponding receives, at each process:
>
> cc = 0
> DO i = 1, my_nbrs
> k = few_entries_from_this_neighbour
> CALL MPI_IRECV(recv_array(cc+1:cc+k),k, MPI_REAL8, source(i), tag,
> MPI_COMM_WORLD, request2(i), ierr)
> DO j = 1, k
> received_data(j) = recv_array(cc+j)
> ENDDO
> cc = cc + k
> ENDDO

Okay, but you're still reading the data before the MPI_Waitall call. If
you call MPI_Irecv(buffer,...), you cannot read the buffer's contents
until the corresponding MPI_Waitall (or variant).

> After the above MPI_WAITALL.
>
> Means that,
> send_array for all neighbours will have a collected shape:
> send_array = [... entries for nbr 1 ..., ... entries for nbr 1 ...,
> ..., ... entries for last nbr ...]
> And the respective entries will be send to respective neighbours as
> above.
>
>
> recv_array for all neighbours will have a collected shape:
> recv_array = [... entries from nbr 1 ..., ... entries from nbr 1 ...,
> ..., ... entries from last nbr ...]
> And the entries from the processes will be received at respective
> locations/portion in the recv_array.
>
>
> Is this scheme is quite fine and correct.
>
> I am in search of efficient one.