Open MPI logo

Open MPI Development Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Development mailing list

Subject: Re: [OMPI devel] Thread safety levels
From: Kawashima (t-kawashima_at_[hidden])
Date: 2010-05-10 03:45:30


Hi Open MPI developers,

Though Sylvain's original mail (*1) was sent 4 months ago and nobody
replied to it, I'm interested in this issue and strongly agree with
Sylvain.

 *1 http://www.open-mpi.org/community/lists/devel/2010/01/7275.php

As explained by Sylvain, current Open MPI implementation always returns
MPI_THREAD_SINGLE as provided thread level if neither --enable-mpi-threads
nor --enable-progress-threads was specified at configure (v1.4).

If we use OpenMP with MPI, we need at least MPI_THREAD_FUNNELED even
if MPI functions are called only outside of omp parallel region,
like below.

    #pragma omp parallel for
    for (...) {
        /* computation */
    }
    MPI_Allreduce(...);
    #pragma omp parallel for
    for (...) {
        /* computation */
    }

This means Open MPI users must specify --enable-mpi-threads or
--enable-progress-threads to use OpenMP. Is it true?
But this two configure options, i.e. OMPI_HAVE_THREAD_SUPPORT macro,
lead to performance penalty by mutex lock/unlock.

# I know that we can actually use OpenMP with MPI_THREAD_SINGLE.
# I'm talking about reason why Open MPI returns MPI_THREAD_SINGLE.

I believe OMPI_HAVE_THREADS (not OMPI_HAVE_THREAD_SUPPORT !) is sufficient
to support MPI_THREAD_FUNNELED and MPI_THREAD_SERIALIZED, and therefore
OMPI_HAVE_THREAD_SUPPORT should be OMPI_HAVE_THREADS at following
part in ompi_mpi_init function, as suggested by Sylvain.

    ompi_mpi_thread_requested = requested;
    if (OMPI_HAVE_THREAD_SUPPORT == 0) { /* <- should be OMPI_HAVE_THREADS ? */
        ompi_mpi_thread_provided = *provided = MPI_THREAD_SINGLE;
        ompi_mpi_main_thread = NULL;
    } else if (OMPI_ENABLE_MPI_THREADS == 1) {
        ompi_mpi_thread_provided = *provided = requested;
        ompi_mpi_main_thread = opal_thread_get_self();
    } else {
        if (MPI_THREAD_MULTIPLE == requested) {
            ompi_mpi_thread_provided = *provided = MPI_THREAD_SERIALIZED;
        } else {
            ompi_mpi_thread_provided = *provided = requested;
        }
        ompi_mpi_main_thread = opal_thread_get_self();
    }

If not, what is the problem to support MPI_THREAD_FUNNELED and
MPI_THREAD_SERIALIZED without OMPI_HAVE_THREAD_SUPPORT?

Though options and code above are those of v1.4, trunk seems to same.
MPI_THREAD_MULTIPLE needs --enable-opal-multi-threads or
--enable-mpi-thread-multiple and they lead to performance penalty.

Regards,
Kawashima

> Hi list,
>
> I'm currently playing with thread levels in Open MPI and I'm quite
> surprised by the current code.
>
> First, the C interface :
> at ompi/mpi/c/init_thread.c:56 we have :
> #if OPAL_ENABLE_MPI_THREADS
> *provided = MPI_THREAD_MULTIPLE;
> #else
> *provided = MPI_THREAD_SINGLE;
> #endif
> prior to the call to ompi_mpi_init() which will in turn override the
> "provided" value. Should we remove these 5 lines ?
>
> Then at ompi/runtime/ompi_mpi_init.c:372, we have -I guess- the real code
> which is :
>
> ompi_mpi_thread_requested = requested;
> if (OPAL_HAVE_THREAD_SUPPORT == 0) {
> ompi_mpi_thread_provided = *provided = MPI_THREAD_SINGLE;
> ompi_mpi_main_thread = NULL;
> } else if (OPAL_ENABLE_MPI_THREADS == 1) {
> ompi_mpi_thread_provided = *provided = requested;
> ompi_mpi_main_thread = opal_thread_get_self();
> } else {
> if (MPI_THREAD_MULTIPLE == requested) {
> ompi_mpi_thread_provided = *provided = MPI_THREAD_SERIALIZED;
> } else {
> ompi_mpi_thread_provided = *provided = requested;
> }
> ompi_mpi_main_thread = opal_thread_get_self();
> }
>
> This code seems ok to me provided that :
> * (OPAL_ENABLE_MPI_THREADS == 1) means "Open MPI configured to provide
> thread multiple",
> * (OPAL_HAVE_THREAD_SUPPORT == 0) means "we do not have threads at all"
> though even if we do not have threads at compile time, it does in no way
> prevent us from doing THREAD_FUNNELED or THREAD_SERIALIZED.
>
> The reality seems different at opal/include/opal_config_bottom.h:70 :
>
> /* Do we have posix or solaris thread lib */
> #define OPAL_HAVE_THREADS (OPAL_HAVE_POSIX_THREADS || OPAL_HAVE_SOLARIS_THREADS)
> /* Do we have thread support? */
> #define OPAL_HAVE_THREAD_SUPPORT (OPAL_ENABLE_MPI_THREADS || OPAL_ENABLE_PROGRESS_THREADS)
>
> "we do not have threads at all" seems to me to be OPAL_HAVE_THREADS and
> not OPAL_HAVE_THREAD_SUPPORT. What do you think ? Maybe
> OPAL_HAVE_THREAD_SUPPORT should be renamed, too (seems misleading to me).
>
> The result is that the current default configuration of Open MPI has
> OPAL_HAVE_THREAD_SUPPORT defined to 0 and Open MPI always returns
> THREAD_SINGLE, even if it is perfectly capable of THREAD_FUNNELED and
> THREAD_SERIALIZED.
>
> Sylvain