Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: [OMPI users] Coding help requested
From: amjad ali (amjad11_at_[hidden])
Date: 2009-11-10 13:25:38


Hi all.
(sorry for duplication, if it is)

I have to parallelize a CFD code using domain/grid/mesh partitioning among
the processes. Before running, we do not know,
(i) How many processes we will use ( np is unknown)
(ii) A process will have how many neighbouring processes (my_nbrs = ?)
(iii) How many entries a process need to send to a particular neighbouring
process.
But when the code run, I calculate all of this info easily.

The problem is to copy a number of entries to an array then send that array
to a destination process. The same sender has to repeat this work to send
data to all of its neighbouring processes. Is this following code fine:

DO i = 1, my_nbrs
   DO j = 1, few_entries_for_this_neighbour
       send_array(j) = my_array(jth_particular_entry)
   ENDDO
   CALL MPI_ISEND(send_array(1:j),j, MPI_REAL8, dest(i), tag,
MPI_COMM_WORLD, request1(i), ierr)
ENDDO

And the corresponding receives, at each process:

DO i = 1, my_nbrs
   k = few_entries_from_this_neighbour
   CALL MPI_IRECV(recv_array(1:k),k, MPI_REAL8, source(i), tag,
MPI_COMM_WORLD, request2(i), ierr)
   DO j = 1, few_from_source(i)
       received_data(j) = recv_array(j)
   ENDDO
ENDDO

After the above MPI_WAITALL.

I think this code will not work. Both for sending and receiving. For the
non-blocking sends we cannot use send_array to send data to other processes
like above (as we are not sure for the availability of application buffer
for reuse). Am I right?

Similar problem is with recv array; data from multiple processes cannot be
received in the same array like above. Am I right?

Target is to hide communication behind computation. So need non blocking
communication. As we do know value of np or values of my_nbrs for each
process, we cannot decide to create so many arrays. Please suggest solution.

===================
A more subtle solution that I could assume is following:

cc = 0
DO i = 1, my_nbrs
   DO j = 1, few_entries_for_this_neighbour
       send_array(cc+j) = my_array(jth_particular_entry)
   ENDDO
   CALL MPI_ISEND(send_array(cc:cc+j),j, MPI_REAL8, dest(i), tag,
MPI_COMM_WORLD, request1(i), ierr)
   cc = cc + j
ENDDO

And the corresponding receives, at each process:

cc = 0
DO i = 1, my_nbrs
   k = few_entries_from_this_neighbour
   CALL MPI_IRECV(recv_array(cc+1:cc+k),k, MPI_REAL8, source(i), tag,
MPI_COMM_WORLD, request2(i), ierr)
   DO j = 1, k
       received_data(j) = recv_array(cc+j)
   ENDDO
   cc = cc + k
ENDDO

After the above MPI_WAITALL.

Means that,
send_array for all neighbours will have a collected shape:
send_array = [... entries for nbr 1 ..., ... entries for nbr 1 ..., ..., ...
entries for last nbr ...]
And the respective entries will be send to respective neighbours as above.

recv_array for all neighbours will have a collected shape:
recv_array = [... entries from nbr 1 ..., ... entries from nbr 1 ..., ...,
... entries from last nbr ...]
And the entries from the processes will be received at respective
locations/portion in the recv_array.

Is this scheme is quite fine and correct.

I am in search of efficient one.

Request for help.

With best regards,
Amjad Ali.