Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: [OMPI users] Problem with MPI_Scatter() on inter-communicator...
From: Audet, Martin (Martin.Audet_at_[hidden])
Date: 2008-04-04 12:18:58


Hi,

I don't know if it is my sample code or if it is a problem whit MPI_Scatter() on inter-communicator (maybe similar to the problem we found with MPI_Allgather() on inter-communicator a few weeks ago) but a simple program I wrote freeze during its second iteration of a loop doing an MPI_Scatter() over an inter-communicator.

For example if I compile as follows:

  mpicc -Wall scatter_bug.c -o scatter_bug

I get no error or warning. Then if a start it with np=2 as follows:

    mpiexec -n 2 ./scatter_bug

it prints:

   beginning Scatter i_root_group=0
   ending Scatter i_root_group=0
   beginning Scatter i_root_group=1

and then hang...

Note also that if I change the for loop to execute only the MPI_Scatter() of the second iteration (e.g. replacing "i_root_group=0;" by "i_root_group=1;"), it prints:

    beginning Scatter i_root_group=1

and then hang...

The problem therefore seems to be related with the second iteration itself.

Please note that this program run fine with mpich2 1.0.7rc2 (ch3:sock device) for many different number of process (np) when the executable is ran with or without valgrind.

The OpenMPI version I use is 1.2.6rc3 and was configured as follows:

   ./configure --prefix=/usr/local/openmpi-1.2.6rc3 --disable-mpi-f77 --disable-mpi-f90 --disable-mpi-cxx --disable-cxx-exceptions --with-io-romio-flags=--with-file-system=ufs+nfs

Note also that all process (when using OpenMPI or mpich2) were started on the same machine.

Also if you look at source code, you will notice that some arguments to MPI_Scatter() are NULL or 0. This may look strange and problematic when using a normal intra-communicator. However according to the book "MPI - The complete reference" vol 2 about MPI-2, for MPI_Scatter() with an inter-communicator:

  "The sendbuf, sendcount and sendtype arguments are significant only at the root process. The recvbuf, recvcount, and recvtype arguments are significant only at the processes of the leaf group."

If anyone else can have a look at this program and try it it would be helpful.

Thanks,

Martin

#include <stdio.h>
#include <stdlib.h>
#include <mpi.h>

int main(int argc, char **argv)
{
   int ret_code = 0;
   int comm_size, comm_rank;

   MPI_Init(&argc, &argv);

   MPI_Comm_size(MPI_COMM_WORLD, &comm_size);
   MPI_Comm_rank(MPI_COMM_WORLD, &comm_rank);

   if (comm_size > 1) {
      MPI_Comm subcomm, intercomm;
      const int group_id = comm_rank % 2;
      int i_root_group;

      /* split process in two groups: even and odd comm_ranks. */
      MPI_Comm_split(MPI_COMM_WORLD, group_id, 0, &subcomm);

      /* The remote leader comm_rank for even and odd groups are respectively: 1 and 0 */
      MPI_Intercomm_create(subcomm, 0, MPI_COMM_WORLD, 1-group_id, 0, &intercomm);

      /* for i_root_group==0 process with comm_rank==0 scatter data to all process with odd comm_rank */
      /* for i_root_group==1 process with comm_rank==1 scatter data to all process with even comm_rank */
      for (i_root_group=0; i_root_group < 2; i_root_group++) {
         if (comm_rank == 0) {
            printf("beginning Scatter i_root_group=%d\n",i_root_group);
         }
         if (group_id == i_root_group) {
            const int is_root = (comm_rank == i_root_group);
            int *send_buf = NULL;
            if (is_root) {
               const int nbr_other = (comm_size+i_root_group)/2;
               int ii;
               send_buf = malloc(nbr_other*sizeof(*send_buf));
               for (ii=0; ii < nbr_other; ii++) {
                   send_buf[ii] = ii;
               }
            }
            MPI_Scatter(send_buf, 1, MPI_INT,
                        NULL, 0, MPI_INT, (is_root ? MPI_ROOT : MPI_PROC_NULL), intercomm);

            if (is_root) {
               free(send_buf);
            }
         }
         else {
            int an_int;
            MPI_Scatter(NULL, 0, MPI_INT,
                        &an_int, 1, MPI_INT, 0, intercomm);
         }
         if (comm_rank == 0) {
            printf("ending Scatter i_root_group=%d\n",i_root_group);
         }
      }

      MPI_Comm_free(&intercomm);
      MPI_Comm_free(&subcomm);
   }
   else {
      fprintf(stderr, "%s: error this program must be started np > 1\n", argv[0]);
      ret_code = 1;
   }

   MPI_Finalize();

   return ret_code;
}