Open MPI logo

Open MPI Development Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Development mailing list

Subject: Re: [OMPI devel] MPI_THREAD_MULTIPLE testing on trunk
From: Christopher Yeoh (cyeoh_at_[hidden])
Date: 2012-08-08 09:55:51


> On Thu, 2 Aug 2012 10:25:53 -0400
> Jeff Squyres <jsquyres_at_[hidden]> wrote:
>
> > On Aug 1, 2012, at 9:44 AM, Christopher Yeoh wrote:
> >
> (gdb) bt
> #0 0x0000008039720d6c in .pthread_cond_wait ()
> from /lib64/power6/libpthread.so.0 #1 0x00000400001299d8 in
> opal_condition_wait (c=0x400004763f8, m=0x40000476460)
> at ../../ompi-trunk.chris2/opal/threads/condition.h:79 #2
> 0x000004000012a08c in ompi_request_default_wait_all (count=2,
> requests=0xfffffa9db20, statuses=0x0)
> at ../../ompi-trunk.chris2/ompi/request/req_wait.c:281 #3
> 0x000004000012f56c in ompi_init_preconnect_mpi ()
> at ../../ompi-trunk.chris2/ompi/runtime/ompi_mpi_preconnect.c:72 #4
> 0x000004000012c738 in ompi_mpi_init (argc=1, argv=0xfffffa9f278,
> requested=3, provided=0xfffffa9edd8)
> at ../../ompi-trunk.chris2/ompi/runtime/ompi_mpi_init.c:800 #5
> 0x000004000017a064 in PMPI_Init_thread (argc=0xfffffa9ee20,
> argv=0xfffffa9ee28, required=3, provided=0xfffffa9edd8) at
> pinit_thread.c:84 #6 0x0000000010000ae4 in main (argc=1,
> argv=0xfffffa9f278) at test2.c:15
>
> (neither of the requests are received so presumably messages are
> getting lost).
>
> In contrast if you run against the exact same build of OMPI with
> pretty much the same test program but do "MPI_Init(&argc, &argv)"
> then it works fine.
>

I think I've worked out what is going on. The difference between 1.6 and
trunk is that in 1.6 is the #ifdefs in opal/threads/condition.h and how
they are set by configure. Its a bit complicated because there was some
renaming done between 1.6 and trunk.

An excerpt from opal/threads/condition.h from 1.6 (in
opal_condition_wait):

    if (opal_using_threads()) {
#if OPAL_HAVE_POSIX_THREADS && OPAL_ENABLE_PROGRESS_THREADS
        rc = pthread_cond_wait(&c->c_cond, &m->m_lock_pthread);
#elif OPAL_HAVE_SOLARIS_THREADS && OPAL_ENABLE_PROGRESS_THREADS
        rc = cond_wait(&c->c_cond, &m->m_lock_solaris);
#else
             if (c->c_signaled) {
            c->c_waiting--;
            opal_mutex_unlock(m);
            opal_progress();

and from trunk:

    if (opal_using_threads()) {
#if OPAL_HAVE_POSIX_THREADS && OPAL_ENABLE_MULTI_THREADS
        rc = pthread_cond_wait(&c->c_cond, &m->m_lock_pthread);
#elif OPAL_HAVE_SOLARIS_THREADS && OPAL_ENABLE_MULTI_THREADS
        rc = cond_wait(&c->c_cond, &m->m_lock_solaris);
#else
        if (c->c_signaled) {
            c->c_waiting--;
            opal_mutex_unlock(m);
            opal_progress();

Now in 1.6 OPAL_ENABLE_PROGRESS_THREADS is hardcoded by configure to be
off. So even with mpi threads enabled when we are in
ompi_request_default_wait_all and call opal_condition_wait we still
call opal_progress.

In trunk OPAL_ENABLE_MULTI_THREADS is set to 1 if mpi threads are
enabled. Note that in 1.6 OPAL_ENABLE_MULTI_THREADS also exists and is
set to 1 if mpi threads are enabled, but as can be seen above is not
used to control how opal_condition_wait behaves.

So in trunk when MPI_THREAD_MULTIPLE is requrest in init, the
pthread_cond_wait path is taken. MPI programs get stuck because the
main thread sits in pthread_cond_wait and there appears to be no one
around to call opal_progress. I've looked around in the OMPI code to see
where a thread should be spawned to service opal_progress, but I
haven't been able to find it.

Between 1.6 and trunk OPAL_ENABLE_PROGRESS_THREADS seems to have
disappeared and OMPI_ENABLE_PROGRESS_THREADS has appeared. The latter
is hardcoded to be off. I tried to compile with
OMPI_ENABLE_PROGRESS_THREADS set, but there are compile errors
(presumably why its turned off). But I'm wondering if in
opal_condition_wait and a few other areas if OPAL_ENABLE_MULTI_THREADS
should in fact be OMPI_ENABLE_PROGRESS_THREADS?

If I change a few of those OPAL_ENABLE_MULTI_THREADS to
OMPI_ENABLE_PROGRESS_THREADS (I don't know if I changed all that need
to be changed) then I can start running threaded MPI programs again.

Regards,

Chris

-- 
cyeoh_at_[hidden]