Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: Re: [OMPI users] Factor of 10 loss in performance with 1.3.x
From: Ethan Mallove (ethan.mallove_at_[hidden])
Date: 2009-04-07 15:22:56


Hi Steve,

I see improvements in 1.3.1 as compared to 1.2.9 in Netpipe results.
The below Open MPI installations were compiled with the same compiler,
configure options, run on the same cluster, and run with the same MCA
parameters. (Note, ClusterTools 8.2 is essentially
1.3.1r20828.)

  http://www.open-mpi.org/mtt/index.php?do_redir=990

Though the above are just two selected Netpipe runs out of thousands
in the MTT database. It's possible with some digging, one could come
up with a couple runs showing 1.2.9 outperforming 1.3.1 (given a
certain set of MCA params, configure options, platform, etc).

-Ethan

On Mon, Apr/06/2009 09:33:46AM, Steve Kargl wrote:
> Hi,
>
> I recently upgraded OpenMPI from 1.2.9 to 1.3 and then 1.3.1.
> One of my colleagues reported a dramatic drop in performance
> with one of his applications. My investigation shows a factor
> of 10 drop in communication over the memory bus. I've placed
> a figure that iilustrates the problem at
>
> http://troutmask.apl.washington.edu/~kargl/ompi_cmp.jpg
>
> The legend in the figure has 'ver. 1.2.9 11 <--> 18'. This
> means communication between node 11 and node 18 over GigE
> ethernet in my cluster. 'ver. 1.2.9 20 <--> 20' means
> communication between processes on node 20 where node 20 has
> 8 processors. The image clearly shows that communication over
> GigE is consistent among the versions of OpenMPI. However, some
> change in going from 1.2.9 to 1.3.x is causing a drop in
> communication between processes on a single node.
>
> Things to note. Nodes 11, 18, and 20 are essentially idle
> before and after a test. configure was run with the same set
> of options except with 1.3 and 1.3.1 I needed to disable ipv6:
>
> ./configure --prefix=/usr/local/openmpi-1.2.9 \
> --enable-orterun-prefix-by-default --enable-static
> --disable-shared
>
> ./configure --prefix=/usr/local/openmpi-1.3.1 \
> --enable-orterun-prefix-by-default --enable-static
> --disable-shared --disable-ipv6
>
> ./configure --prefix=/usr/local/openmpi-1.3.1 \
> --enable-orterun-prefix-by-default --enable-static
> --disable-shared --disable-ipv6
>
> The operating system is FreeBSD 8.0 where nodes 18 and 20
> are quad-core, dual-cpu opteron based systems and node 11
> is a dual-core, dual-cpu opteron based system. For additional
> information, I've placed the output of ompi_info at
>
> http://troutmask.apl.washington.edu/~kargl/ompi_info-1.2.9
> http://troutmask.apl.washington.edu/~kargl/ompi_info-1.3.0
> http://troutmask.apl.washington.edu/~kargl/ompi_info-1.3.1
>
> Any hints on tuning 1.3.1 would be appreciated?
>
> --
> steve
> --
> Steve
> _______________________________________________
> users mailing list
> users_at_[hidden]
> http://www.open-mpi.org/mailman/listinfo.cgi/users