Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |  

This web mail archive is frozen.

This page is part of a frozen web archive of this mailing list.

You can still navigate around this archive, but know that no new mails have been added to it since July of 2016.

Click here to be taken to the new web archives of this list; it includes all the mails that are in this frozen archive plus all new mails that have been sent to the list since it was migrated to the new archives.

Subject: Re: [OMPI users] performance of MPI_Iallgatherv
From: Zehan Cui (zehan.cui_at_[hidden])
Date: 2014-04-08 21:29:48


Thanks, it looks like I have to do the overlapping myself.

On Tue, Apr 8, 2014 at 5:40 PM, Matthieu Brucher <matthieu.brucher_at_[hidden]
> wrote:

> Yes, usually the MPI libraries don't allow that. You can launch
> another thread for the computation, make calls to MPI_Test during that
> time and join at the end.
>
> Cheers,
>
> 2014-04-07 4:12 GMT+01:00 Zehan Cui <zehan.cui_at_[hidden]>:
> > Hi Matthieu,
> >
> > Thanks for your suggestion. I tried MPI_Waitall(), but the results are
> > the same. It seems the communication didn't overlap with computation.
> >
> > Regards,
> > Zehan
> >
> > On 4/5/14, Matthieu Brucher <matthieu.brucher_at_[hidden]> wrote:
> >> Hi,
> >>
> >> Try waiting on all gathers at the same time, not one by one (this is
> >> what non blocking collectives are made for!)
> >>
> >> Cheers,
> >>
> >> Matthieu
> >>
> >> 2014-04-05 10:35 GMT+01:00 Zehan Cui <zehan.cui_at_[hidden]>:
> >>> Hi,
> >>>
> >>> I'm testing the non-blocking collective of OpenMPI-1.8.
> >>>
> >>> I have two nodes with Infiniband to perform allgather on totally 128MB
> >>> data.
> >>>
> >>> I split the 128MB data into eight pieces, and perform computation and
> >>> MPI_Iallgatherv() on one piece of data each iteration, hoping that the
> >>> MPI_Iallgatherv() of last iteration can be overlapped with computation
> of
> >>> current iteration. A MPI_Wait() is called at the end of last iteration.
> >>>
> >>> However, the total communication time (including the final wait time)
> is
> >>> similar with that of the traditional blocking MPI_Allgatherv, even
> >>> slightly
> >>> higher.
> >>>
> >>>
> >>> Following is the test pseudo-code, the source code are attached.
> >>>
> >>> ===========================
> >>>
> >>> Using MPI_Allgatherv:
> >>>
> >>> for( i=0; i<8; i++ )
> >>> {
> >>> // computation
> >>> mytime( t_begin );
> >>> computation;
> >>> mytime( t_end );
> >>> comp_time += (t_end - t_begin);
> >>>
> >>> // communication
> >>> t_begin = t_end;
> >>> MPI_Allgatherv();
> >>> mytime( t_end );
> >>> comm_time += (t_end - t_begin);
> >>> }
> >>> --------------------------------------------
> >>>
> >>> Using MPI_Iallgatherv:
> >>>
> >>> for( i=0; i<8; i++ )
> >>> {
> >>> // computation
> >>> mytime( t_begin );
> >>> computation;
> >>> mytime( t_end );
> >>> comp_time += (t_end - t_begin);
> >>>
> >>> // communication
> >>> t_begin = t_end;
> >>> MPI_Iallgatherv();
> >>> mytime( t_end );
> >>> comm_time += (t_end - t_begin);
> >>> }
> >>>
> >>> // wait for non-blocking allgather to complete
> >>> mytime( t_begin );
> >>> for( i=0; i<8; i++ )
> >>> MPI_Wait;
> >>> mytime( t_end );
> >>> wait_time = t_end - t_begin;
> >>>
> >>> ==============================
> >>>
> >>> The results of Allgatherv is:
> >>> [cmy_at_gnode102 test_nbc]$ /home3/cmy/czh/opt/ompi-1.8/bin/mpirun -n 2
> >>> --host
> >>> gnode102,gnode103 ./Allgatherv 128 2 | grep time
> >>> Computation time : 8481279 us
> >>> Communication time: 319803 us
> >>>
> >>> The results of Iallgatherv is:
> >>> [cmy_at_gnode102 test_nbc]$ /home3/cmy/czh/opt/ompi-1.8/bin/mpirun -n 2
> >>> --host
> >>> gnode102,gnode103 ./Iallgatherv 128 2 | grep time
> >>> Computation time : 8479177 us
> >>> Communication time: 199046 us
> >>> Wait time: 139841 us
> >>>
> >>>
> >>> So, does this mean that current OpenMPI implementation of
> MPI_Iallgatherv
> >>> doesn't support offloading of collective communication to dedicated
> cores
> >>> or
> >>> network interface?
> >>>
> >>> Best regards,
> >>> Zehan
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> _______________________________________________
> >>> users mailing list
> >>> users_at_[hidden]
> >>> http://www.open-mpi.org/mailman/listinfo.cgi/users
> >>
> >>
> >>
> >> --
> >> Information System Engineer, Ph.D.
> >> Blog: http://matt.eifelle.com
> >> LinkedIn: http://www.linkedin.com/in/matthieubrucher
> >> Music band: http://liliejay.com/
> >> _______________________________________________
> >> users mailing list
> >> users_at_[hidden]
> >> http://www.open-mpi.org/mailman/listinfo.cgi/users
> >>
> >
> >
> > --
> > Best Regards
> > Zehan Cui(崔泽汉)
> > -----------------------------------------------------------
> > Institute of Computing Technology, Chinese Academy of Sciences.
> > No.6 Kexueyuan South Road Zhongguancun,Haidian District Beijing,China
> > _______________________________________________
> > users mailing list
> > users_at_[hidden]
> > http://www.open-mpi.org/mailman/listinfo.cgi/users
>
>
>
> --
> Information System Engineer, Ph.D.
> Blog: http://matt.eifelle.com
> LinkedIn: http://www.linkedin.com/in/matthieubrucher
> Music band: http://liliejay.com/
> _______________________________________________
> users mailing list
> users_at_[hidden]
> http://www.open-mpi.org/mailman/listinfo.cgi/users
>