Open MPI logo

Open MPI Development Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Development mailing list

Subject: Re: [OMPI devel] Thread safety levels
From: N.M. Maclaren (nmm1_at_[hidden])
Date: 2010-05-10 05:26:15


>On May 10 2010, Kawashima wrote:
>
>Though Sylvain's original mail (*1) was sent 4 months ago and nobody
>replied to it, I'm interested in this issue and strongly agree with
>Sylvain.
>
> *1 http://www.open-mpi.org/community/lists/devel/2010/01/7275.php
>
>As explained by Sylvain, current Open MPI implementation always returns
>MPI_THREAD_SINGLE as provided thread level if neither --enable-mpi-threads
>nor --enable-progress-threads was specified at configure (v1.4).

I can explain that, from an outside viewpoint. I can't tell you why
OpenMPI took that decision, but I can guess.

That is definitely the correct action. Unless an application or library
has been built with thread support, or can guaranteed to be called only
from a single thread, using threads is catastrophic. And, regrettably,
given modern approaches to building software and the **** configure
design, configure is where the test has to go.

On some systems, there are certain actions that require thread affinity
(sometimes including I/O, and often undocumented). zOS is one, but I
have seen it under a few Unices, too.

On others, they use a completely different (and seriously incompatible,
at both the syntactic and semantic levels) set of libraries. E.g. AIX.

>If we use OpenMP with MPI, we need at least MPI_THREAD_FUNNELED even
>if MPI functions are called only outside of omp parallel region,
>like below.
>
> #pragma omp parallel for
> for (...) {
> /* computation */
> }
> MPI_Allreduce(...);
> #pragma omp parallel for
> for (...) {
> /* computation */
> }

I don't think that's correct. That would call MPI_Allreduce once for
each thread, in parallel, on the same process - which wouldn't work.
I think that what you need is a primitive that OpenMP doesn't have (in
general), which is a GLOBAL_MASTER construct. What you have to do is:

    Each process finds its initial (system) thread id on entry.
    You test the system thread and call MPI only if on that one.

>This means Open MPI users must specify --enable-mpi-threads or
>--enable-progress-threads to use OpenMP. Is it true?
>But this two configure options, i.e. OMPI_HAVE_THREAD_SUPPORT macro,
>lead to performance penalty by mutex lock/unlock.

That's unavoidable, in general, with one niggle. If the programmer
guarantees BOTH to call MPI on the global master thread AND to ensure
that all memory is synchronised before it does so, there is no need
for mutexes. The MPI specification lacks some of the necessary
paranoia in this respect.

>I believe OMPI_HAVE_THREADS (not OMPI_HAVE_THREAD_SUPPORT !) is sufficient
>to support MPI_THREAD_FUNNELED and MPI_THREAD_SERIALIZED, and therefore
>OMPI_HAVE_THREAD_SUPPORT should be OMPI_HAVE_THREADS at following
>part in ompi_mpi_init function, as suggested by Sylvain.

I can't comment on that, though I doubt it's quite that simple. There's
a big difference between MPI_THREAD_FUNNELED and MPI_THREAD_SERIALIZED
in implementation impact.

Regards,
Nick Maclaren.