Open MPI logo

Open MPI Development Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Development mailing list

Subject: [OMPI devel] Different behaviour with MPI_IN_PLACE in MPI_Reduce_scatter() and MPI_Ireduce_scatter()
From: Lisandro Dalcin (dalcinl_at_[hidden])
Date: 2014-04-21 10:05:48


I'm not sure this is actually a bug, but the difference may surprise
users. It seems that the implementation of
MPI_Ireduce_scatter(MPI_IN_PLACE,...) (ab?)uses the recvbuf to compute
the intermediate reduction, while MPI_Reduce_scatter(MPI_IN_PLACE,...)
does not.

Look at the following code (setup to be run in up to 16 processes).
While MPI_Reduce_scatter() does not change the second and following
elements of recvbuf, the nonblocking variant do modify the second and
following entries in some ranks.

[dalcinl_at_kw2060 openmpi]$ cat ireduce_scatter.c
#include <stdlib.h>
#include <stdio.h>
#include <mpi.h>
int main(int argc, char *argv[])
{
  int i,size,rank;
  int recvbuf[] = {1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1};
  int rcounts[] = {1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1};
  MPI_Init(&argc, &argv);
  MPI_Comm_size(MPI_COMM_WORLD, &size);
  MPI_Comm_rank(MPI_COMM_WORLD, &rank);
  if (size > 16) MPI_Abort(MPI_COMM_WORLD,1);
#ifndef NBCOLL
#define NBCOLL 1
#endif
#if NBCOLL
  {
    MPI_Request request;
    MPI_Ireduce_scatter(MPI_IN_PLACE, recvbuf, rcounts, MPI_INT,
                        MPI_SUM, MPI_COMM_WORLD, &request);
    MPI_Wait(&request,MPI_STATUS_IGNORE);
  }
#else
  MPI_Reduce_scatter(MPI_IN_PLACE, recvbuf, rcounts, MPI_INT,
                     MPI_SUM, MPI_COMM_WORLD);
#endif
  printf("[%d] rbuf[%d]=%2d expected:%2d\n", rank, 0, recvbuf[i], size);
  for (i=1; i<size; i++) {
    printf("[%d] rbuf[%d]=%2d expected:%2d\n", rank, i, recvbuf[i], 1);
  }
  MPI_Finalize();
  return 0;
}

[dalcinl_at_kw2060 openmpi]$ mpicc -DNBCOLL=0 ireduce_scatter.c
[dalcinl_at_kw2060 openmpi]$ mpiexec -n 3 ./a.out | sort
[0] rbuf[0]= 3 expected: 3
[0] rbuf[1]= 1 expected: 1
[0] rbuf[2]= 1 expected: 1
[1] rbuf[0]= 3 expected: 3
[1] rbuf[1]= 1 expected: 1
[1] rbuf[2]= 1 expected: 1
[2] rbuf[0]= 3 expected: 3
[2] rbuf[1]= 1 expected: 1
[2] rbuf[2]= 1 expected: 1

[dalcinl_at_kw2060 openmpi]$ mpicc -DNBCOLL=1 ireduce_scatter.c
[dalcinl_at_kw2060 openmpi]$ mpiexec -n 3 ./a.out | sort
[0] rbuf[0]= 3 expected: 3
[0] rbuf[1]= 2 expected: 1
[0] rbuf[2]= 2 expected: 1
[1] rbuf[0]= 3 expected: 3
[1] rbuf[1]= 1 expected: 1
[1] rbuf[2]= 1 expected: 1
[2] rbuf[0]= 3 expected: 3
[2] rbuf[1]= 1 expected: 1
[2] rbuf[2]= 1 expected: 1

-- 
Lisandro Dalcin
---------------
CIMEC (UNL/CONICET)
Predio CONICET-Santa Fe
Colectora RN 168 Km 472, Paraje El Pozo
3000 Santa Fe, Argentina
Tel: +54-342-4511594 (ext 1016)
Tel/Fax: +54-342-4511169