Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: Re: [OMPI users] scaling problem with openmpi
From: Peter Kjellstrom (cap_at_[hidden])
Date: 2009-05-20 13:12:55

On Wednesday 20 May 2009, Pavel Shamis (Pasha) wrote:
> > Disabling basic_linear seems like a good idea but your config file sets
> > the cut-off at 128 Bytes for 64-ranks (the field you set to 8192 seems to
> > result in a message size of that value divided by the number of ranks).
> >
> > In my testing bruck seems to win clearly (at least for 64 ranks on my IB)
> > up to 2048. Hence, the following line may be better:
> >
> > 131072 2 0 0 # switch to pair wise for size 128K/nranks
> >
> > Disclaimer: I guess this could differ quite a bit for nranks!=64 and
> > different btls.
> Sounds strange for me. From the code is looks that we take the threshold as
> is without dividing by number of ranks.

Interesting, I may have had to little or too much coffe but the figures in my
previous e-mail (3rd run, bruckto2k_pair) was run with the above line. And it
very much looks like it switched at 128K/64=2K, not at 128K (which would have
been above my largest size of 3000 and as such equiv. to all_bruck).

I also ran tests with:
 8192 2 0 0 # ...
And it seemed to switch between 10 Bytes and 500 Bytes (most likely then at

My testprogram calls MPI_Alltoall like this:
  time1 = MPI_Wtime();
  for (i = 0; i < repetitions; i++) {
    MPI_Alltoall(sbuf, message_size, MPI_CHAR,
                 rbuf, message_size, MPI_CHAR, MPI_COMM_WORLD);
  time2 = MPI_Wtime();