Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: Re: [OMPI users] scaling problem with openmpi
From: Roman Martonak (r.martonak_at_[hidden])
Date: 2009-05-20 14:27:22


I tried to run with the first dynamic rules file that Pavel proposed
and it works, the time per one MD step on 48 cores decreased from 2.8
s to 1.8 s as expected. It was clearly the basic linear algorithm that
was causing the problem. I will check the performance of bruck and
pairwise on my HW. It would be nice if it could be tuned further.

Thanks

Roman

On Wed, May 20, 2009 at 7:18 PM, Pavel Shamis (Pasha) <pashash_at_[hidden]> wrote:
> Tomorrow I will add some printf to collective code and check what really
> happens there...
>
> Pasha
>
> Peter Kjellstrom wrote:
>>
>> On Wednesday 20 May 2009, Pavel Shamis (Pasha) wrote:
>>
>>>>
>>>> Disabling basic_linear seems like a good idea but your config file sets
>>>> the cut-off at 128 Bytes for 64-ranks (the field you set to 8192 seems
>>>> to
>>>> result in a message size of that value divided by the number of ranks).
>>>>
>>>> In my testing bruck seems to win clearly (at least for 64 ranks on my
>>>> IB)
>>>> up to 2048. Hence, the following line may be better:
>>>>
>>>>  131072 2 0 0 # switch to pair wise for size 128K/nranks
>>>>
>>>> Disclaimer: I guess this could differ quite a bit for nranks!=64 and
>>>> different btls.
>>>>
>>>
>>> Sounds strange for me. From the code is looks that we take the threshold
>>> as
>>> is without dividing by number of ranks.
>>>
>>
>> Interesting, I may have had to little or too much coffe but the figures in
>> my previous e-mail (3rd run, bruckto2k_pair) was run with the above line.
>> And it very much looks like it switched at 128K/64=2K, not at 128K (which
>> would have been above my largest size of 3000 and as such equiv. to
>> all_bruck).
>>
>> I also ran tests with:
>>  8192 2 0 0 # ...
>> And it seemed to switch between 10 Bytes and 500 Bytes (most likely then
>> at 8192/64=128).
>>
>> My testprogram calls MPI_Alltoall like this:
>>  time1 = MPI_Wtime();
>>  for (i = 0; i < repetitions; i++) {
>>    MPI_Alltoall(sbuf, message_size, MPI_CHAR,
>>                 rbuf, message_size, MPI_CHAR, MPI_COMM_WORLD);
>>  }
>>  time2 = MPI_Wtime();
>>
>> /Peter
>>  ------------------------------------------------------------------------
>>
>> _______________________________________________
>> users mailing list
>> users_at_[hidden]
>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>
> _______________________________________________
> users mailing list
> users_at_[hidden]
> http://www.open-mpi.org/mailman/listinfo.cgi/users
>