Open MPI logo

Open MPI Development Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Development mailing list

Subject: Re: [OMPI devel] Thread safety levels
From: Sylvain Jeaugey (sylvain.jeaugey_at_[hidden])
Date: 2010-05-10 07:04:27


On Mon, 10 May 2010, N.M. Maclaren wrote:

>> As explained by Sylvain, current Open MPI implementation always returns
>> MPI_THREAD_SINGLE as provided thread level if neither --enable-mpi-threads
>> nor --enable-progress-threads was specified at configure (v1.4).
>
> That is definitely the correct action. Unless an application or library
> has been built with thread support, or can guaranteed to be called only
> from a single thread, using threads is catastrophic.
I personnaly see that as a bug, but I certainly lack some knowledge on
non-linux OSes. From my point of view, any normal library should be
THREAD_SERIALIZED, and thread-safe library should be THREAD_MULTIPLE. I
don't see other libraries which claims to be "totally incompatible with
the use of threads". They may not be thread-safe, in which case the
programmer must ensure locking and memory coherency to use them in
conjunction with threads, but that's about what THREAD_SERIALIZED is about
IMO.

> And, regrettably,
> given modern approaches to building software and the **** configure
> design, configure is where the test has to go.
configure is where the tests is. And configure indeed returns "We have
threads" (OMPI_HAVE_THREADS = 1). And given this, I don't see why we
wouldn't be MPI_THREAD_SERIALIZED. At least MPI_THREAD_FUNELLED.

> On some systems, there are certain actions that require thread affinity
> (sometimes including I/O, and often undocumented). zOS is one, but I
> have seen it under a few Unices, too.
>
> On others, they use a completely different (and seriously incompatible,
> at both the syntactic and semantic levels) set of libraries. E.g. AIX.
>
>> If we use OpenMP with MPI, we need at least MPI_THREAD_FUNNELED even
>> if MPI functions are called only outside of omp parallel region,
>> like below.
>>
>> #pragma omp parallel for
>> for (...) {
>> /* computation */
>> }
>> MPI_Allreduce(...);
>> #pragma omp parallel for
>> for (...) {
>> /* computation */
>> }
>
> I don't think that's correct. That would call MPI_Allreduce once for
> each thread, in parallel, on the same process - which wouldn't work.
I think the idea is precisely _not_ to call MPI_Allreduce within parallel
sections, i.e. only have the master thread call MPI.

>> This means Open MPI users must specify --enable-mpi-threads or
>> --enable-progress-threads to use OpenMP. Is it true?
>> But this two configure options, i.e. OMPI_HAVE_THREAD_SUPPORT macro,
>> lead to performance penalty by mutex lock/unlock.
>
> That's unavoidable, in general, with one niggle. If the programmer
> guarantees BOTH to call MPI on the global master thread AND to ensure
> that all memory is synchronised before it does so, there is no need
> for mutexes. The MPI specification lacks some of the necessary
> paranoia in this respect.
In my understanding of MPI_THREAD_SERIALIZED, the memory coherency was
guaranteed. If not, the programmer has to ensure it.

>> I believe OMPI_HAVE_THREADS (not OMPI_HAVE_THREAD_SUPPORT !) is sufficient
>> to support MPI_THREAD_FUNNELED and MPI_THREAD_SERIALIZED, and therefore
>> OMPI_HAVE_THREAD_SUPPORT should be OMPI_HAVE_THREADS at following
>> part in ompi_mpi_init function, as suggested by Sylvain.
>
> I can't comment on that, though I doubt it's quite that simple. There's
> a big difference between MPI_THREAD_FUNNELED and MPI_THREAD_SERIALIZED
> in implementation impact.
I don't see the relationship between THREAD_SERIALIZED/FUNNELED and
OMPI_HAVE_THREAD_SUPPORT. Actually, OMPI_HAVE_THREAD_SUPPORT seems to have
no relationship with how the OS supports threads (that's why I think it is
misleading).

But I don't see a big difference between THREAD_SERIALIZED and
THREAD_FUNNELED anyway. Do you have more information on systems where the
caller thread id makes a difference in MPI ?

Just for the record, we (at Bull) patched our MPI library and had no
problem so far with applications using MPI + Threads or MPI + OpenMP,
given that they don't call MPI within parallel sections. But of course, we
only use linux, so your mileage may vary.

Sylvain