Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: Re: [OMPI users] scaling problem with openmpi
From: Peter Kjellstrom (cap_at_[hidden])
Date: 2009-05-20 11:55:54


On Wednesday 20 May 2009, Pavel Shamis (Pasha) wrote:
> > With the file Pavel has provided things have changed to the following.
> > (maybe someone can confirm)
> >
> > If message size < 8192
> > bruck
> > else
> > pairwise
> > end
>
> You are right here. Target of my conf file is disable basic_linear for
> medium message side.

Disabling basic_linear seems like a good idea but your config file sets the
cut-off at 128 Bytes for 64-ranks (the field you set to 8192 seems to result
in a message size of that value divided by the number of ranks).

In my testing bruck seems to win clearly (at least for 64 ranks on my IB) up
to 2048. Hence, the following line may be better:

 131072 2 0 0 # switch to pair wise for size 128K/nranks

Disclaimer: I guess this could differ quite a bit for nranks!=64 and different
btls.

Here are some figures for this part of the package size range:

all_bruck
bw for 10 x 10 B : 13.7 Mbytes/s time was: 922.0 �s
bw for 10 x 500 B : 45.9 Mbytes/s time was: 13.7 ms
bw for 10 x 1000 B : 122.7 Mbytes/s time was: 10.3 ms
bw for 10 x 1500 B : 86.9 Mbytes/s time was: 21.8 ms
bw for 10 x 2000 B : 120.1 Mbytes/s time was: 21.0 ms
bw for 10 x 2047 B : 92.6 Mbytes/s time was: 27.9 ms
bw for 10 x 2048 B : 107.3 Mbytes/s time was: 24.1 ms
bw for 10 x 2400 B : 93.7 Mbytes/s time was: 32.3 ms
bw for 10 x 2800 B : 73.0 Mbytes/s time was: 48.3 ms
bw for 10 x 2900 B : 79.5 Mbytes/s time was: 45.9 ms
bw for 10 x 2925 B : 89.3 Mbytes/s time was: 41.3 ms
bw for 10 x 2950 B : 72.7 Mbytes/s time was: 51.1 ms
bw for 10 x 2975 B : 75.2 Mbytes/s time was: 49.8 ms
bw for 10 x 3000 B : 74.9 Mbytes/s time was: 50.5 ms
bw for 10 x 3100 B : 95.9 Mbytes/s time was: 40.7 ms
totaltime was: 479.5 ms
all_pair
bw for 10 x 10 B : 414.2 kbytes/s time was: 30.4 ms
bw for 10 x 500 B : 19.8 Mbytes/s time was: 31.9 ms
bw for 10 x 1000 B : 43.3 Mbytes/s time was: 29.1 ms
bw for 10 x 1500 B : 63.3 Mbytes/s time was: 29.9 ms
bw for 10 x 2000 B : 81.2 Mbytes/s time was: 31.0 ms
bw for 10 x 2047 B : 82.3 Mbytes/s time was: 31.3 ms
bw for 10 x 2048 B : 83.0 Mbytes/s time was: 31.1 ms
bw for 10 x 2400 B : 93.6 Mbytes/s time was: 32.3 ms
bw for 10 x 2800 B : 105.0 Mbytes/s time was: 33.6 ms
bw for 10 x 2900 B : 107.7 Mbytes/s time was: 33.9 ms
bw for 10 x 2925 B : 108.1 Mbytes/s time was: 34.1 ms
bw for 10 x 2950 B : 109.6 Mbytes/s time was: 33.9 ms
bw for 10 x 2975 B : 111.1 Mbytes/s time was: 33.7 ms
bw for 10 x 3000 B : 112.1 Mbytes/s time was: 33.7 ms
bw for 10 x 3100 B : 114.5 Mbytes/s time was: 34.1 ms
totaltime was: 484.1 ms
bruckto2k_pair
bw for 10 x 10 B : 11.9 Mbytes/s time was: 1.1 ms
bw for 10 x 500 B : 100.3 Mbytes/s time was: 6.3 ms
bw for 10 x 1000 B : 115.9 Mbytes/s time was: 10.9 ms
bw for 10 x 1500 B : 117.2 Mbytes/s time was: 16.1 ms
bw for 10 x 2000 B : 95.7 Mbytes/s time was: 26.3 ms
bw for 10 x 2047 B : 96.6 Mbytes/s time was: 26.7 ms
bw for 10 x 2048 B : 82.2 Mbytes/s time was: 31.4 ms
bw for 10 x 2400 B : 94.1 Mbytes/s time was: 32.1 ms
bw for 10 x 2800 B : 105.6 Mbytes/s time was: 33.4 ms
bw for 10 x 2900 B : 108.4 Mbytes/s time was: 33.7 ms
bw for 10 x 2925 B : 108.3 Mbytes/s time was: 34.0 ms
bw for 10 x 2950 B : 109.9 Mbytes/s time was: 33.8 ms
bw for 10 x 2975 B : 111.5 Mbytes/s time was: 33.6 ms
bw for 10 x 3000 B : 108.3 Mbytes/s time was: 34.9 ms
bw for 10 x 3100 B : 114.7 Mbytes/s time was: 34.0 ms
totaltime was: 388.4 ms

These figures were run on a freshly compiled OpenMPI-1.3.2. The numbers for
bruck at smalla package sizes vary a bit from run to run.

/Peter

> Pasha.