Bonsoir Eugene,

 First thanks for trying to help me.

 I already gave a try to some profiling tool, namely IPM, which is rather
simple to use. Here follows some output for a 1024 core run.
Unfortunately, I'm yet unable to have the equivalent MPT chart.

#IPMv0.983####################################################################
#
# command : unknown (completed)
# host    : r34i0n0/x86_64_Linux           mpi_tasks : 1024 on 128 nodes
# start   : 12/21/10/13:18:09              wallclock : 3357.308618 sec
# stop    : 12/21/10/14:14:06              %comm     : 27.67
# gbytes  : 0.00000e+00 total              gflop/sec : 0.00000e+00 total
#
##############################################################################
# region  : *       [ntasks] =   1024
#
#                           [total]         <avg>           min           max
# entries                       1024             1             1             1
# wallclock              3.43754e+06       3356.98       3356.83       3357.31
# user                   2.82831e+06       2762.02       2622.04       2923.37
# system                      376230       367.412       174.603       492.919
# mpi                         951328       929.031       633.137       1052.86
# %comm                                    27.6719       18.8601        31.363
# gflop/sec                        0             0             0             0
# gbytes                           0             0             0             0
#
#
#                            [time]       [calls]        <%mpi>      <%wall>
# MPI_Waitall                 741683   7.91081e+07         77.96        21.58
# MPI_Allreduce               114057   2.53665e+07         11.99         3.32
# MPI_Recv                   40164.7          2048          4.22         1.17
# MPI_Isend                  27420.6   6.53513e+08          2.88         0.80
# MPI_Barrier                25113.5          2048          2.64         0.73
# MPI_Sendrecv                2123.6        212992          0.22         0.06
# MPI_Irecv                  464.616   6.53513e+08          0.05         0.01
# MPI_Reduce                 215.447        171008          0.02         0.01
# MPI_Bcast                  85.0198          1024          0.01         0.00
# MPI_Send                  0.377043          2048          0.00         0.00
# MPI_Comm_rank          0.000744925          4096          0.00         0.00
# MPI_Comm_size          0.000252183          1024          0.00         0.00
###############################################################################

 It seems to my non-expert eye that MPI_Waitall is dominant among MPI calls,
but not for the overall application, however I will have to compare with MPT,
before concluding.

 Thanks again for your suggestions, that I'll address one by one.

 Best,     G.

 


Le 22/12/2010 18:50, Eugene Loh a écrit :
Can you isolate a bit more where the time is being spent?  The performance effect you're describing appears to be drastic.  Have you profiled the code?  Some choices of tools can be found in the FAQ http://www.open-mpi.org/faq/?category=perftools  The results may be "uninteresting" (all time spent in your MPI_Waitall calls, for example), but it'd be good to rule out other possibilities (e.g., I've seen cases where it's the non-MPI time that's the culprit).

If all the time is spent in MPI_Waitall, then I wonder if it would be possible for you to reproduce the problem with just some MPI_Isend|Irecv|Waitall calls that mimic your program.  E.g., "lots of short messages", or "lots of long messages", etc.  It sounds like there is some repeated set of MPI exchanges, so maybe that set can be extracted and run without the complexities of the application.

Anyhow, some profiling might help guide one to the problem.

Gilbert Grosdidier wrote:

There are indeed a high rate of communications. But the buffer
size is always the same for a given pair of processes, and I thought
that mpi_leave_pinned should avoid freeing the memory in this case.
Am I wrong ?