Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

From: Graham E Fagg (fagg_at_[hidden])
Date: 2006-01-04 12:39:57


Thanks Carsten,
  I have started updating my jumpshot so will let you know as soon as I
have some ideas on whats going on.
G.
ps. I am going offline now for 2 days while travelling

On Wed, 4 Jan 2006, Carsten Kutzner wrote:

> Hi Graham,
>
> here are the all-to-all test results with the modification to the decision
> routine you suggested yesterday. Now the routine behaves nicely for 128
> and 256 float messages on 128 CPUs! For the other sizes one probably wants
> to keep the original algorithm, since it is faster there. However I have
> the feeling that for messages >= 4096 floats there still exists the old
> problem since the execution times are so variable there (note that the
> standard deviation rises by more than a factor of 10 when going from 2048
> to 4096 floats.) If you need additional test results to tune the decision
> functions please let me know.
>
> Carsten
>
>
> OMPI tuned all-to-all with modification:
> ========================================
> mesg size time in seconds
> #CPUs floats average std.dev. min. max.
> 128 1 0.001253 0.000074 0.001141 0.001470
> 128 2 0.023507 0.000563 0.022562 0.024761
> 128 4 0.023435 0.000426 0.022582 0.024166
> 128 8 0.023438 0.000359 0.022904 0.024104
> 128 16 0.023664 0.000438 0.022844 0.024670
> 128 32 0.024136 0.000463 0.023297 0.025117
> 128 64 0.024704 0.000535 0.023727 0.026030
> 128 128 0.025750 0.000525 0.024592 0.026799 *
> 128 256 0.028862 0.000683 0.027389 0.030168 *
> 128 512 0.035869 0.001214 0.034067 0.038655
> 128 1024 0.046528 0.001722 0.043549 0.050432
> 128 2048 0.072388 0.007032 0.066708 0.104358
> 128 4096 0.217678 0.097312 0.135113 0.409431
> 128 8192 0.378586 0.090267 0.297878 0.577771
> 128 16384 0.567473 0.105083 0.483573 0.735509
> 128 32768 1.151343 0.146547 0.937150 1.404478
> 128 65536 2.298998 0.169669 1.983286 2.572027
> 128 131072 4.070989 0.159958 3.691039 4.373587
>
>
>
>>> OMPI tuned all-to-all:
>>> ======================
>>> mesg size time in seconds
>>> #CPUs floats average std.dev. min. max.
>>> 128 1 0.001288 0.000102 0.001077 0.001512
>>> 128 2 0.008391 0.000400 0.007861 0.009958
>>> 128 4 0.008403 0.000237 0.008095 0.009018
>>> 128 8 0.008228 0.000942 0.003801 0.008810
>>> 128 16 0.008503 0.000191 0.008233 0.008839
>>> 128 32 0.008656 0.000271 0.008084 0.009177
>>> 128 64 0.009085 0.000209 0.008757 0.009603
>>> 128 128 0.251414 0.073069 0.011547 0.506703 !
>>> 128 256 0.385515 0.127661 0.251431 0.578955 !
>>> 128 512 0.035111 0.000872 0.033358 0.036262
>>> 128 1024 0.046028 0.002116 0.043381 0.052602
>>> 128 2048 0.073392 0.007745 0.066432 0.104531
>>> 128 4096 0.165052 0.072889 0.124589 0.404213
>>> 128 8192 0.341377 0.041815 0.309457 0.530409
>>> 128 16384 0.507200 0.050872 0.492307 0.750956
>>> 128 32768 1.050291 0.132867 0.954496 1.344978
>>> 128 65536 2.213977 0.154987 1.962907 2.492560
>>> 128 131072 4.026107 0.147103 3.800191 4.336205
>>>
>>> alternative all-to-all:
>>> ======================
>>> 128 1 0.012584 0.000724 0.011073 0.015331
>>> 128 2 0.012506 0.000444 0.011707 0.013461
>>> 128 4 0.012412 0.000511 0.011157 0.013413
>>> 128 8 0.012488 0.000455 0.011767 0.013746
>>> 128 16 0.012664 0.000416 0.011745 0.013362
>>> 128 32 0.012878 0.000410 0.012157 0.013609
>>> 128 64 0.013138 0.000417 0.012452 0.013826
>>> 128 128 0.014016 0.000505 0.013195 0.014942 +
>>> 128 256 0.015843 0.000521 0.015107 0.016725 +
>>> 128 512 0.052240 0.079323 0.027019 0.320653 !
>>> 128 1024 0.123884 0.121560 0.038062 0.308929 !
>>> 128 2048 0.176877 0.125229 0.074457 0.387276 !
>>> 128 4096 0.305030 0.121716 0.176640 0.496375 !
>>> 128 8192 0.546405 0.108007 0.415272 0.899858 !
>>> 128 16384 0.604844 0.056576 0.558657 0.843943 !
>>> 128 32768 1.235298 0.097969 1.094720 1.451241 !
>>> 128 65536 2.926902 0.312733 2.458742 3.895563 !
>>> 128 131072 6.208087 0.472115 5.354304 7.317153 !
>
>
> ---------------------------------------------------
> Dr. Carsten Kutzner
> Max Planck Institute for Biophysical Chemistry
> Theoretical and Computational Biophysics Department
> Am Fassberg 11
> 37077 Goettingen, Germany
> Tel. +49-551-2012313, Fax: +49-551-2012302
> eMail ckutzne_at_[hidden]
> http://www.gwdg.de/~ckutzne
>
>

Thanks,
         Graham.
----------------------------------------------------------------------
Dr Graham E. Fagg | Distributed, Parallel and Meta-Computing
Innovative Computing Lab. PVM3.4, HARNESS, FT-MPI, SNIPE & Open MPI
Computer Science Dept | Suite 203, 1122 Volunteer Blvd,
University of Tennessee | Knoxville, Tennessee, USA. TN 37996-3450
Email: fagg_at_[hidden] | Phone:+1(865)974-5790 | Fax:+1(865)974-8296
Broken complex systems are always derived from working simple systems
----------------------------------------------------------------------