Open MPI logo

Open MPI Development Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Development mailing list

Subject: Re: [OMPI devel] Thread safety levels
From: N.M. Maclaren (nmm1_at_[hidden])
Date: 2010-05-10 08:49:14

On May 10 2010, Sylvain Jeaugey wrote:
>> That is definitely the correct action. Unless an application or library
>> has been built with thread support, or can guaranteed to be called only
>> from a single thread, using threads is catastrophic.
>I personnaly see that as a bug, but I certainly lack some knowledge on
>non-linux OSes.

It's not a bug, except possibly in Linux. Threading is optional in
POSIX and not all MPI hosts are Unices, anyway. It would be
reasonable for OpenMPI to demand a certain minimum level of threading
support, given that non-threaded systems are more-or-less dead.

>From my point of view, any normal library should be
>THREAD_SERIALIZED, and thread-safe library should be THREAD_MULTIPLE.

I am not disagreeing, but that's a matter of the system designer's

>I don't see other libraries which claims to be "totally incompatible with
>the use of threads". They may not be thread-safe, in which case the
>programmer must ensure locking and memory coherency to use them in
>conjunction with threads, but that's about what THREAD_SERIALIZED is about

No, that's not what I meant. There really are systems out there where
you must compile with a threading option to ensure that threads can be
supported. AIX is (or, at least, was) one - and a right pain it was,
too! I had to edit the compile scripts to get them to work with IBM's
own MPI - and I didn't support the MPI+OpenMP mixture, either.

>> I don't think that's correct. That would call MPI_Allreduce once for
>> each thread, in parallel, on the same process - which wouldn't work.
>I think the idea is precisely _not_ to call MPI_Allreduce within parallel
>sections, i.e. only have the master thread call MPI.

Then you need the extra code I mentioned, but that doesn't affect
your main point. Let's ignore this one.

>In my understanding of MPI_THREAD_SERIALIZED, the memory coherency was
>guaranteed. If not, the programmer has to ensure it.

It can't guarantee it when running under a POSIX-like system; the
programmer has to ensure it. There is no one-sided synchronisation
mechanism in POSIX.

>> I can't comment on that, though I doubt it's quite that simple. There's
>> a big difference between MPI_THREAD_FUNNELED and MPI_THREAD_SERIALIZED
>> in implementation impact.
>I don't see the relationship between THREAD_SERIALIZED/FUNNELED and
>no relationship with how the OS supports threads (that's why I think it is

Only someone who knows the history of OpenMPI's configure mechanism
could answer that. I can't.

>But I don't see a big difference between THREAD_SERIALIZED and
>THREAD_FUNNELED anyway. Do you have more information on systems where the
>caller thread id makes a difference in MPI ?

MPI isn't the issue - it's the underlying facilities. For example,
some uses of sockets might fail if they weren't on the same thread
as the one that opened the socket. That was a documented feature of
MVS (now zOS) and I have seen it on several Unices. POSIX doesn't
specify this properly, and says several things that add unnecessary
confusion. There is also a potential problem with InfiniBand and
similar transports, but that's tricky to explain.

I don't know how serious this problem is, today, on OpenMPI's target
systems, but I know that it's a real problem and truly evil when it
occurs. That is, after all, why MPI distinguishes those cases.

>Just for the record, we (at Bull) patched our MPI library and had no
>problem so far with applications using MPI + Threads or MPI + OpenMP,
>given that they don't call MPI within parallel sections. But of course, we
>only use linux, so your mileage may vary.

That doesn't surprise me. Linux is usually free from such gotchas.

Nick Maclaren.