Doing Reduce without Barrier first allows one process to call Reduce and exit immediately without waiting for other processes to call Reduce.  Therefore, this allows one process to advance faster than other processes.  I suspect the 2671 second result is the difference between the fastest and slowest process.  Having Barrier reduce the time difference because it forces the faster processes to go slower.

On Wed, Sep 8, 2010 at 3:21 AM, Gabriele Fatigati <g.fatigati@cineca.it> wrote:
Dear OpenMPI users,

i'm using OpenMPI 1.3.3 on Infiniband 4x interconnnection network. My parallel application use intensive MPI_Reduce communication over communicator created with MPI_Comm_split.

I've noted strange behaviour during execution. My code is instrumented with Scalasca 1.3 to report subroutine execution time. First execution shows elapsed time with 128 processors ( job_communicator is created with MPI_Comm_split). In both cases is composed to the same ranks of MPI_COMM_WORLD:

MPI_Reduce(.....,job_communicator)

The elapsed time is 2671 sec.

Second run use MPI_BARRIER before MPI_Reduce:

MPI_Barrier(job_communicator..)
MPI_Reduce(.....,job_communicator)

The elapsed time of Barrier+Reduce is 2167 sec, (about 8 minutes less). 

So, im my opinion, it is better put MPI_Barrier before any MPI_Reduce to mitigate "asynchronous" behaviour of MPI_Reduce in OpenMPI. I suspect the same for others collective communications. Someone can explaine me why MPI_reduce has this strange behaviour?

Thanks in advance.




--
Ing. Gabriele Fatigati

Parallel programmer

CINECA Systems & Tecnologies Department

Supercomputing Group

Via Magnanelli 6/3, Casalecchio di Reno (BO) Italy

www.cineca.it                    Tel:   +39 051 6171722

g.fatigati [AT] cineca.it          

_______________________________________________
users mailing list
users@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users



--
David Zhang
University of California, San Diego