Open MPI logo

Open MPI Development Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Development mailing list

Subject: Re: [OMPI devel] Finalize without Detach???
From: George Bosilca (bosilca_at_[hidden])
Date: 2009-11-18 17:47:10


The proper practice based on the MPI Standard will be to call the
detach function before finalize. From a pure OMPI perspective, we do
the same thing in both cases, i.e. we wait until all pending
communications on the buffer are completed to detach it. I think dhe
difference in performance came from the fact that in case of
MPI_Finalize we call the poll at every iteration in opal_progress.

   george.

On Nov 18, 2009, at 12:18 , Eugene Loh wrote:

> What's the story about calling MPI_Finalize without first calling
> MPI_Buffer_detach?
>
> If I do an MPI_Bsend followed by MPI_Finalize, the corresponding
> MPI_Recv takes forever. In contrast, if I insert an
> MPI_Buffer_detach, then performance is reasonable. I can imagine
> the explanation. I suspect that MPI_Bsend leaves the message in a
> local buffer, and so you need to progress the sender in order for
> the receive to complete. MPI_Buffer_detach must progress more
> aggressively than MPI_Finalize.
>
> 1) Any guidance from MPI gurus regarding what is proper practice?
> 2) Any guidance from OMPI devels what sort of fix makes sense?
>
> I attach a test case. On some platforms, the final delay can be on
> order of a minute.
>
> % mpif90 main.F90
> % mpirun -np 2 -mca btl sm,self a.out
> 1 0.021321
> 0 0.066568
> 1 0.020978
> 0 0.061625
> 1 0.021969
> 0 0.062380
> 1 0.020938
> 0 0.064401
> 1 0.020759
> 0 4.098010 # yipes! last receive takes a long time!
> % mpif90 -DDETACH main.F90
> % mpirun -np 2 -mca btl sm,self a.out
> 1 0.020913
> 0 0.064076
> 1 0.020746
> 0 0.061015
> 1 0.020454
> 0 0.061780
> 1 0.020457
> 0 0.060776
> 1 0.020619
> 0 0.062484
> include "mpif.h"
> integer, parameter :: nbufbytes = 16000000, nsendbytes = 15892480
> real(8) buf(nbufbytes/8), x(nsendbytes/8), t
> real(8) buf2
> integer mbufbytes
>
> call MPI_Init(ier)
> call MPI_Comm_size(MPI_COMM_WORLD,np,ier)
> call MPI_Comm_rank(MPI_COMM_WORLD,me,ier)
> buf = 0.d0
> x = 0.d0
>
> if ( me == 1 ) call MPI_Buffer_attach(buf, nbufbytes, ier)
>
> do i = 1, 5
> call MPI_Barrier(MPI_COMM_WORLD,ier)
> t = MPI_Wtime()
> if ( me == 0 ) call MPI_Recv (x,nsendbytes,MPI_BYTE,
> 1,343,MPI_COMM_WORLD,MPI_STATUS_IGNORE,ier)
> if ( me == 1 ) call MPI_Bsend(x,nsendbytes,MPI_BYTE,
> 0,343,MPI_COMM_WORLD, ier)
> t = MPI_Wtime() - t
> write(6,'(i4,f12.6,f8.3)') me, t
> end do
>
> #ifdef DETACH
> if ( me == 1 ) call MPI_Buffer_detach(buf2, mbufbytes, ier)
> #endif
>
> call MPI_Finalize(ier)
> end
> _______________________________________________
> devel mailing list
> devel_at_[hidden]
> http://www.open-mpi.org/mailman/listinfo.cgi/devel