Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: Re: [OMPI users] MPI_Init_thread hangs in OpenMPI 1.7.1 when using --enable-mpi-thread-multiple
From: Ralph Castain (rhc_at_[hidden])
Date: 2013-10-23 18:41:15


Yes, I expect it will continue to hang until 1.7.4 is released. Should be
fixed there, though we'll check.

On Wed, Oct 23, 2013 at 11:11 AM, Paul Kapinos <kapinos_at_[hidden]>wrote:

> Just kindly reminder: this bug seem still to exist in 1.7.3 :-/
>
>
>
> On 08/20/13 22:15, Ralph Castain wrote:
>
>> On Aug 20, 2013, at 12:40 PM, RoboBeans <robobeans_at_[hidden]
>> <mailto:robobeans_at_[hidden]>> wrote:
>>
>> I can confirm that mpi program still hangs if one uses these options
>>> while
>>> configuring openmpi-1.7.2
>>>
>>> --enable-event-thread-support --enable-opal-multi-threads
>>> --enable-orte-progress-threads --enable-mpi-thread-multiple
>>>
>>>
>>> *cd /opt**
>>> **gunzip -c openmpi-1.7.2.tar.gz | tar xf -**
>>> **cd openmpi-1.7.2**
>>> **./configure --with-openib=/usr --with-psm=/usr
>>> --prefix=/opt/openmpi-1.7.2
>>> --enable-event-thread-support --enable-opal-multi-threads
>>> --enable-orte-progress-threads --enable-mpi-thread-multiple**
>>> **make all install**
>>> **cd ..**
>>> **chmod -R 775 openmpi-1.7.2**
>>> **ldconfig*
>>>
>>>
>> Yep, sure will
>>
>>
>> Is there any plan to support multiple threads any time soon?
>>>
>>
>> Not in the immediate future, but hopefully not too far into infinity -
>> however,
>> we should fix the hang in the near future, at least
>>
>>
>>> *$ uname -r*
>>>
>>>
>>> 2.6.32-358.14.1.el6.x86_64
>>>
>>>
>>> Thanks!
>>>
>>> On 6/19/13 12:51 PM, Hans Ekkehard Plesser wrote:
>>>
>>>> I can confirm that
>>>>
>>>> --enable-event-thread-support --enable-opal-multi-threads
>>>> --enable-orte-progress-threads --enable-mpi-thread-multiple
>>>>
>>>> worked for me as well under OSX 10.8.4 with clang. The orte-progress
>>>> flag is needed, without it the parallel process hangs.
>>>>
>>>> I will report this to the MacPorts people so they can fix it.
>>>>
>>>> Best,
>>>> Hans
>>>>
>>>> On 19. juni 2013, at 15.08, Ralph Castain wrote:
>>>>
>>>> While I'm glad it worked, having it depend on orte progress thread
>>>>> wasn't done by intent, hence the comment in the configure help output. I'm
>>>>> not seeing any problems with using that option, but please let us know if
>>>>> you run into something. It hasn't been as fully tested as we'd like.
>>>>>
>>>>>
>>>>> On Jun 19, 2013, at 12:10 AM, Elias Rudberg<elias.rudberg_at_[hidden]**>
>>>>> wrote:
>>>>>
>>>>> This has now been solved.
>>>>>>
>>>>>> The solution (as suggested by Ralph) is to use the following four
>>>>>> configure options together:
>>>>>> ./configure --enable_event_thread_support --enable_opal_multi_threads
>>>>>> --enable_orte_progress_threads --enable_mpi_thread_multiple
>>>>>>
>>>>>> So, apparently the --enable_mpi_thread_multiple option cannot be used
>>>>>> by itself, it must be combined with the other three options, otherwise
>>>>>> MPI_Init_thread will hang.
>>>>>>
>>>>>> / Elias
>>>>>>
>>>>>>
>>>>>> Quoting Elias Rudberg<elias.rudberg_at_[hidden]**>:
>>>>>>
>>>>>> Hi,
>>>>>>>
>>>>>>> I now tried using --enable-opal-multi-threads (I added an "s" in the
>>>>>>> end, since "--enable-opal-multi-thread" was not recognized by the configure
>>>>>>> script).
>>>>>>>
>>>>>>> However, my test code still hangs in the same way as before.
>>>>>>>
>>>>>>> To be clear, I now configure like this:
>>>>>>> ./configure --enable-mpi-thread-multiple --enable-opal-multi-threads
>>>>>>>
>>>>>>> but the problem remains: MPI_Init_thread still hangs when trying to
>>>>>>> run the test case with -np 2.
>>>>>>>
>>>>>>> Any other ideas?
>>>>>>> Are there other configure options that must be used together with
>>>>>>> --enable-mpi-thread-multiple to make it work?
>>>>>>>
>>>>>>> Best regards,
>>>>>>> Elias
>>>>>>>
>>>>>>>
>>>>>>> Quoting Ralph Castain<rhc_at_[hidden]>:
>>>>>>>
>>>>>>> Hmmm...well, your code runs fine for me:
>>>>>>>>
>>>>>>>> Ralphs-iMac:mpi rhc$ mpirun -n 2 ./thread_init
>>>>>>>> Calling MPI_Init_thread...
>>>>>>>> Calling MPI_Init_thread...
>>>>>>>> MPI_Init_thread returned, provided = 3
>>>>>>>> MPI_Init_thread returned, provided = 3
>>>>>>>> Ralphs-iMac:mpi rhc$
>>>>>>>>
>>>>>>>> I think the key, however, is that you also have to configure with
>>>>>>>> --enable-opal-multi-thread.
>>>>>>>>
>>>>>>>>
>>>>>>>> On Jun 16, 2013, at 8:54 AM, Elias Rudberg<elias.rudberg_at_[hidden]**>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>> Hello!
>>>>>>>>>
>>>>>>>>> I would like to report what seems to be a bug in MPI_Init_thread
>>>>>>>>> in OpenMPI 1.7.1.
>>>>>>>>>
>>>>>>>>> The bug can be reproduced with the following test program
>>>>>>>>> (test_mpi_thread_support.c):
>>>>>>>>> ==============================**=============
>>>>>>>>> #include <mpi.h>
>>>>>>>>> #include <stdio.h>
>>>>>>>>> int main(int argc, const char* argv[]) {
>>>>>>>>> int provided = -1;
>>>>>>>>> printf("Calling MPI_Init_thread...\n");
>>>>>>>>> MPI_Init_thread(NULL, NULL, MPI_THREAD_MULTIPLE, &provided);
>>>>>>>>> printf("MPI_Init_thread returned, provided = %d\n", provided);
>>>>>>>>> MPI_Finalize();
>>>>>>>>> return 0;
>>>>>>>>> }
>>>>>>>>> ==============================**=============
>>>>>>>>>
>>>>>>>>> When trying to run this when OpenMPI was configured with
>>>>>>>>> --enable-mpi-thread-multiple, the program hangs when trying to run using
>>>>>>>>> anything more than one process.
>>>>>>>>>
>>>>>>>>> Steps I use to reproduce this in Ubuntu:
>>>>>>>>>
>>>>>>>>> (1) Download openmpi-1.7.1.tar.gz
>>>>>>>>>
>>>>>>>>> (2) Configure like this:
>>>>>>>>> ./configure --enable-mpi-thread-multiple
>>>>>>>>>
>>>>>>>>> (3) make
>>>>>>>>>
>>>>>>>>> (4) Compile test program like this:
>>>>>>>>> mpicc test_mpi_thread_support.c
>>>>>>>>>
>>>>>>>>> (5) Run like this:
>>>>>>>>> mpirun -np 2 ./a.out
>>>>>>>>> Then you see the following two lines of output:
>>>>>>>>> Calling MPI_Init_thread...
>>>>>>>>> Calling MPI_Init_thread...
>>>>>>>>> And then it hangs.
>>>>>>>>>
>>>>>>>>> MPI_Init_thread did not hang in earlier OpenMPI versions (for
>>>>>>>>> example it worked in 1.5.* and 1.6.*), so it seems like a bug introduced in
>>>>>>>>> 1.7.
>>>>>>>>>
>>>>>>>>> The description above shows how I reproduce this in Ubuntu on my
>>>>>>>>> local desktop computer, but the same problem exists for the OpenMPI 1.7.1
>>>>>>>>> installation at the UPPMAX computer center where I wan to run my code in
>>>>>>>>> the end. I don't know all details about how they installed it there, but I
>>>>>>>>> know they set --enable-mpi-thread-multiple. So maybe it hangs in 1.7.1 on
>>>>>>>>> any computer as long as you use MPI_THREAD_MULTIPLE. At least I have not
>>>>>>>>> seen it work anywhere.
>>>>>>>>>
>>>>>>>>> Do you agree that this is a bug, or am I doing something wrong?
>>>>>>>>>
>>>>>>>>> Best regards,
>>>>>>>>> Elias
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> ______________________________**_________________
>>>>>>>>> users mailing list
>>>>>>>>> users_at_[hidden]
>>>>>>>>> http://www.open-mpi.org/**mailman/listinfo.cgi/users>
>>>>>>>>>
>>>>>>>> ______________________________**_________________
>>>>>>>> users mailing list
>>>>>>>> users_at_[hidden]
>>>>>>>>
http://www.open-mpi.org/**mailman/listinfo.cgi/users>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>> ______________________________**_________________
>>>>> users mailing list
>>>>> users_at_[hidden]
>>>>>
http://www.open-mpi.org/**mailman/listinfo.cgi/users>
>>>>>
>>>> --
>>>>
>>>> Dr. Hans Ekkehard Plesser, Associate Professor
>>>> Head, Basic Science Section
>>>>
>>>> Dept. of Mathematical Sciences and Technology
>>>> Norwegian University of Life Sciences
>>>> PO Box 5003, 1432 Aas, Norway
>>>>
>>>> Phone +47 6496 5467
>>>> Fax +47 6496 5401
>>>> Emailhans.ekkehard.plesser@**umb.no <Emailhans.ekkehard.plesser_at_[hidden]>
>>>> Home
http://arken.umb.no/~**plesser <http://arken.umb.no/~plesser>
>>>>
>>>>
>>>>
>>>> ______________________________**_________________
>>>> users mailing list
>>>> users_at_[hidden]
>>>> http://www.open-mpi.org/**mailman/listinfo.cgi/users>
>>>>
>>>
>>> ______________________________**_________________
>>> users mailing list
>>> users_at_[hidden] <mailto:users_at_[hidden]>
>>>
http://www.open-mpi.org/**mailman/listinfo.cgi/users>
>>>
>>
>>
>>
>> ______________________________**_________________
>> users mailing list
>> users_at_[hidden]
>>
http://www.open-mpi.org/**mailman/listinfo.cgi/users>
>>
>>
>
> --
> Dipl.-Inform. Paul Kapinos - High Performance Computing,
> RWTH Aachen University, Center for Computing and Communication
> Seffenter Weg 23, D 52074 Aachen (Germany)
> Tel: +49 241/80-24915
>
>
> _______________________________________________
> users mailing list
> users_at_[hidden]
>
http://www.open-mpi.org/mailman/listinfo.cgi/users
>