Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: Re: [OMPI users] Problem with HPL while using OpenMPI 1.3.3
From: Gus Correa (gus_at_[hidden])
Date: 2010-01-05 12:31:07


Hi Ilya

1) The only thing that stands out as very different from
what I do here is your configuration flag "--enable-mpi-threads".

Maybe some OpenMPI pro/developer could shed some light about this,
whether that flag could be a potential source for the errors you see.

Considering that when you suppress it all works fine,
I would think that the "--enable-mpi-threads" is the culprit.

2) Another thing is that we never remove the loopback from
the btl, i.e, we do NOT do btl_tcp_if_exclude=lo,eth0,
just btl_tcp_if_exclude=eth0.

I presume your choice will leave only the shared memory "sm"
mechanism for intra-node communication.
Not sure if this is the best choice.

The OpenMPI developers may have something to say about it.

3) Other differences are that we use ssh, not rsh,
but I don't think that rsh would cause a problem to you.
Also, we use Infiniband (IB) instead of Ethernet for production runs.
However, I tested OpenMPI with Ethernet and it works like
a charm too (selecting eth1 just as you did),
although not as fast as with IB.

I hope this helps,
Gus Correa
---------------------------------------------------------------------
Gustavo Correa
Lamont-Doherty Earth Observatory - Columbia University
Palisades, NY, 10964-8000 - USA
---------------------------------------------------------------------

ilya zelenchuk wrote:
> Happy New Year, Gus!
> Yes, I'm using affinity. This is my openmpi-mca-params.conf file:
>
> ---
> # Use RSH instead SSH
> pls_rsh_agent=rsh
>
> # Turning on processor affinity
> mpi_paffinity_alone=1
>
> # Include using eth1.
> btl_tcp_if_include=eth1
>
> # Exclude using lo and eth0.
> btl_tcp_if_exclude=lo,eth0
> ---
>
> I rebuild openmpi-1.3.3 from tar and find something interesting.
> If i build with this options:
>
> CC=icc CXX=icpc F77=ifort FC=ifort ./configure
> --prefix=/opt/testing/default/openmpi-1.3.3/ --disable-ipv6
>
> All work's fine!!!
>
> But if I use:
>
> CC=icc CFLAGS=" -O3" CXX=icpc CXXFLAGS=" -O3" F77=ifort FFLAGS=" -O3"
> FC=ifort FCFLAGS=" -O3" ./configure
> --prefix=/opt/testing/tuning/openmpi-1.3.3/ --enable-debug
> --enable-mpi-threads --disable-ipv6
>
> Then I get errors.
> What you think?
>
> Now I'm starting to investigate, what options cause OpenMPI broken.
> _______________________________________________
> users mailing list
> users_at_[hidden]
> http://www.open-mpi.org/mailman/listinfo.cgi/users