Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

From: Carsten Kutzner (ckutzne_at_[hidden])
Date: 2006-01-04 04:42:44


Hi Peter,

We have HP ProCurve 2848 GigE switches here (48 port). The problem is more
severe the more nodes (=ports) are involved. It starts to show up at 16
ports for a limited range of message sizes and gets really bad for 32
nodes. The switch has a 96 Gbit/s backplane and should therefore be
able to forward the in and out traffic of all 48 ports simultaneously, as
long as not two nodes send to one receiver. The ordered communication pattern
takes care about the latter (e.g. having only pairs communicate at the
same time). Maybe the switch gets problems when switching from one pair to
another? I will try if I can get another switch for testing.

Thanks!
  Carsten

On Wed, 4 Jan 2006, Peter [iso-8859-1] Kjellström wrote:

> Hello Carsten,
>
> Have you considered the possibility that this is the effect of a non-optimal
> ethernet switch? I don't know how many nodes you need to reproduce it on or
> if you even have physical access (and opportunity) but popping in another
> decent 16-port switch for a testrun might be interesting.
>
> just my .02 euros,
> Peter
>
> On Tuesday 03 January 2006 18:45, Carsten Kutzner wrote:
> > On Tue, 3 Jan 2006, Graham E Fagg wrote:
> > > Do you have any tools such as Vampir (or its Intel equivalent) available
> > > to get a time line graph ? (even jumpshot of one of the bad cases such as
> > > the 128/32 for 256 floats below would help).
> >
> > Hi Graham,
> >
> > I have attached an slog file of an all-to-all run for 1024 floats (ompi
> > tuned alltoall). I could not get clog files for >32 processes - is this
> > perhaps a limitation of MPE? So I decided to take the case 32 CPUs on
> > 32 nodes which is performance-critical as well. From the run output you
> > can see that 2 of the 5 tries yield a fast execution while the others
> > are slow (see below).
> >
> > Carsten
> >
> >
> >
> > ckutzne_at_node001:~/mpe> mpirun -hostfile ./bhost1 -np 32 ./phas_mpe.x
> > Alltoall Test on 32 CPUs. 5 repetitions.
> > --- New category (first test not counted) ---
> > MPI: sending 1024 floats ( 4096 bytes) to 32 processes ( 1
> > times) took ... 0.00690 seconds
> > ---------------------------------------------
> > MPI: sending 1024 floats ( 4096 bytes) to 32 processes ( 1
> > times) took ... 0.00320 seconds MPI: sending 1024 floats ( 4096
> > bytes) to 32 processes ( 1 times) took ... 0.26392 seconds ! MPI:
> > sending 1024 floats ( 4096 bytes) to 32 processes ( 1 times)
> > took ... 0.26868 seconds ! MPI: sending 1024 floats ( 4096 bytes)
> > to 32 processes ( 1 times) took ... 0.26398 seconds ! MPI: sending
> > 1024 floats ( 4096 bytes) to 32 processes ( 1 times) took ...
> > 0.00339 seconds Summary (5-run average, timer resolution 0.000001):
> > 1024 floats took 0.160632 (0.143644) seconds. Min: 0.003200 max:
> > 0.268681 Writing logfile....
> > Finished writing logfile.
>
> --
> ------------------------------------------------------------
> Peter Kjellström |
> National Supercomputer Centre |
> Sweden | http://www.nsc.liu.se
>

---------------------------------------------------
Dr. Carsten Kutzner
Max Planck Institute for Biophysical Chemistry
Theoretical and Computational Biophysics Department
Am Fassberg 11
37077 Goettingen, Germany
Tel. +49-551-2012313, Fax: +49-551-2012302
eMail ckutzne_at_[hidden]
http://www.gwdg.de/~ckutzne