Open MPI logo

Open MPI Development Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Development mailing list

Subject: Re: [OMPI devel] Hierarchical Collectives Query
From: Edgar Gabriel (gabriel_at_[hidden])
Date: 2008-01-24 20:02:33


Rolf,

Whoowh! That's actually good news, since in our own tests hierarch is
always slower. But this might be due to various reasons, including the
fact, that we only have two cores per node. BTW: I actually would expect
IMB test to have worse performance for hierarch compared to many other
benchmarks, since the rotating root causes some additional work/overhead.

Rolf Vandevaart wrote:
> I am curious if anyone is doing any work currently on the hierarchical
> collectives. I ask this because I just did some runs on a cluster made
> up of 4 servers with 4 processors per server. I used TCP over IB. I
> was running with np=16 and using the IMB benchmark to test MPI_Bcast.
> What I am seeing is that the hierarchical collectives appear to boost
> performance. The IMB test rotates the root so one could imagine that
> since the hierarchical minimizes internode communication, performance
> increases. See the table at the end of this post with the comparison
> for MPI_Bcast between tuned and hierarchical. This leads me to a few
> other questions.
>
> 1. From what I can tell from the debug messages, we still cannot stack
> the hierarchical on top of the tuned. I know that Brian Barrett did
> some work after the collectives meeting to allow for this, but I could
> not figure out how to get it to work.

actually, this should be possible. We do however experience with the
current trunk some problems, so I can not verify right now. So, just for
the sake of clarity, did you run hierarch on top of tuned or on top of
basic and/or sm?

>
> 2. Enabling the hierarchical collectives causes a massive slowdown
> during MPI_Init. I know it was discussed a little at the collectives
> meeting and it appears that this is still something we need to solve.
> For a simple hello_world, np=4, 2 node cluster, I see around 5 seconds
> to run for tuned collectives, but I see around 18 seconds for
> hierarchical.

yes. A faster, however simpler hierarchy detection is implemented, but
not yet committed.

>
> 3. Apart from the MPI_Init issue, is hierarchical ready to go?

Clearly, the algorithms are very simple in hierarch, but they are still
lacking large-scale testing, so this is something which would have to be
incorporated.

We have also experimented with various other hierarchical algorithms for
bcast, over the last few months. Our overall progress has however been
significantly slower than I hoped. I know however, that various other
groups also have interest in the hierarch component, and might also be
ready to invest some time to bring it up to speed.

Thanks
Edgar

>
> 4. As the nodes get fatter, I assume the need for hierarchical
> will increase, so this may become a larger issue for all of us?
>
> RESULTS FROM TWO RUNS OF IMB-MPI1
>
> #----------------------------------------------------------------
> # Benchmarking Bcast
> # #processes = 16 TUNED HIERARCH
> #----------------------------------------------------------------
> #bytes #repetitions t_avg[usec] t_avg[usec]
> 0 1000 0.11 0.22
> 1 1000 205.97 319.86
> 2 1000 159.23 180.80
> 4 1000 175.32 189.16
> 8 1000 153.10 184.26
> 16 1000 170.98 192.33
> 32 1000 160.69 187.17
> 64 1000 159.75 182.62
> 128 1000 175.47 185.19
> 256 1000 160.77 194.68
> 512 1000 265.45 313.89
> 1024 1000 185.66 215.43
> 2048 1000 815.97 257.37
> 4096 1000 1208.48 442.93
> 8192 1000 1521.23 530.54
> 16384 1000 2357.45 813.44
> 32768 1000 3341.29 1455.78
> 65536 640 6485.70 3387.02
> 131072 320 13488.35 5261.65
> 262144 160 24783.09 10747.28
> 524288 80 50906.06 21817.64
> 1048576 40 95466.82 41397.49
> 2097152 20 180759.72 81319.54
> 4194304 10 322327.71 163274.55
>
>
> =========================
> rolf.vandevaart_at_[hidden]
> 781-442-3043
> =========================
> _______________________________________________
> devel mailing list
> devel_at_[hidden]
> http://www.open-mpi.org/mailman/listinfo.cgi/devel

-- 
Edgar Gabriel
Assistant Professor
Parallel Software Technologies Lab      http://pstl.cs.uh.edu
Department of Computer Science          University of Houston
Philip G. Hoffman Hall, Room 524        Houston, TX-77204, USA
Tel: +1 (713) 743-3857                  Fax: +1 (713) 743-3335