Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: [OMPI users] performance of MPI_Iallgatherv
From: Zehan Cui (zehan.cui_at_[hidden])
Date: 2014-04-05 05:35:56


Hi,

I'm testing the non-blocking collective of OpenMPI-1.8.

I have two nodes with Infiniband to perform allgather on totally 128MB data.

I split the 128MB data into eight pieces, and perform computation and
MPI_Iallgatherv() on one piece of data each iteration, hoping that the
MPI_Iallgatherv() of last iteration can be overlapped with computation of
current iteration. A MPI_Wait() is called at the end of last iteration.

However, the total communication time (including the final wait time) is
similar with that of the traditional blocking MPI_Allgatherv, even slightly
higher.

Following is the test pseudo-code, the source code are attached.

===========================

Using MPI_Allgatherv:

for( i=0; i<8; i++ )
{
  // computation
    mytime( t_begin );
    computation;
    mytime( t_end );
    comp_time += (t_end - t_begin);

  // communication
    t_begin = t_end;
    MPI_Allgatherv();
    mytime( t_end );
    comm_time += (t_end - t_begin);
}
--------------------------------------------

Using MPI_Iallgatherv:

for( i=0; i<8; i++ )
{
  // computation
    mytime( t_begin );
    computation;
    mytime( t_end );
    comp_time += (t_end - t_begin);

  // communication
    t_begin = t_end;
    MPI_Iallgatherv();
    mytime( t_end );
    comm_time += (t_end - t_begin);
}

// wait for non-blocking allgather to complete
mytime( t_begin );
for( i=0; i<8; i++ )
    MPI_Wait;
mytime( t_end );
wait_time = t_end - t_begin;

==============================

The results of Allgatherv is:
[cmy_at_gnode102 test_nbc]$ /home3/cmy/czh/opt/ompi-1.8/bin/mpirun -n 2 --host
gnode102,gnode103 ./Allgatherv 128 2 | grep time
Computation time : 8481279 us
Communication time: 319803 us

The results of Iallgatherv is:
[cmy_at_gnode102 test_nbc]$ /home3/cmy/czh/opt/ompi-1.8/bin/mpirun -n 2 --host
gnode102,gnode103 ./Iallgatherv 128 2 | grep time
Computation time : 8479177 us
Communication time: 199046 us
Wait time: 139841 us

So, does this mean that current OpenMPI implementation of MPI_Iallgatherv
doesn't support offloading of collective communication to dedicated cores
or network interface?

Best regards,
Zehan