Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: Re: [OMPI users] MPI_Init_thread hangs in OpenMPI 1.7.1 when using --enable-mpi-thread-multiple
From: Paul Kapinos (kapinos_at_[hidden])
Date: 2013-10-23 14:11:16


Just kindly reminder: this bug seem still to exist in 1.7.3 :-/

On 08/20/13 22:15, Ralph Castain wrote:
> On Aug 20, 2013, at 12:40 PM, RoboBeans <robobeans_at_[hidden]
> <mailto:robobeans_at_[hidden]>> wrote:
>
>> I can confirm that mpi program still hangs if one uses these options while
>> configuring openmpi-1.7.2
>>
>> --enable-event-thread-support --enable-opal-multi-threads
>> --enable-orte-progress-threads --enable-mpi-thread-multiple
>>
>>
>> *cd /opt**
>> **gunzip -c openmpi-1.7.2.tar.gz | tar xf -**
>> **cd openmpi-1.7.2**
>> **./configure --with-openib=/usr --with-psm=/usr --prefix=/opt/openmpi-1.7.2
>> --enable-event-thread-support --enable-opal-multi-threads
>> --enable-orte-progress-threads --enable-mpi-thread-multiple**
>> **make all install**
>> **cd ..**
>> **chmod -R 775 openmpi-1.7.2**
>> **ldconfig*
>>
>
> Yep, sure will
>
>
>> Is there any plan to support multiple threads any time soon?
>
> Not in the immediate future, but hopefully not too far into infinity - however,
> we should fix the hang in the near future, at least
>
>>
>> *$ uname -r*
>>
>> 2.6.32-358.14.1.el6.x86_64
>>
>>
>> Thanks!
>>
>> On 6/19/13 12:51 PM, Hans Ekkehard Plesser wrote:
>>> I can confirm that
>>>
>>> --enable-event-thread-support --enable-opal-multi-threads --enable-orte-progress-threads --enable-mpi-thread-multiple
>>>
>>> worked for me as well under OSX 10.8.4 with clang. The orte-progress flag is needed, without it the parallel process hangs.
>>>
>>> I will report this to the MacPorts people so they can fix it.
>>>
>>> Best,
>>> Hans
>>>
>>> On 19. juni 2013, at 15.08, Ralph Castain wrote:
>>>
>>>> While I'm glad it worked, having it depend on orte progress thread wasn't done by intent, hence the comment in the configure help output. I'm not seeing any problems with using that option, but please let us know if you run into something. It hasn't been as fully tested as we'd like.
>>>>
>>>>
>>>> On Jun 19, 2013, at 12:10 AM, Elias Rudberg<elias.rudberg_at_[hidden]> wrote:
>>>>
>>>>> This has now been solved.
>>>>>
>>>>> The solution (as suggested by Ralph) is to use the following four configure options together:
>>>>> ./configure --enable_event_thread_support --enable_opal_multi_threads --enable_orte_progress_threads --enable_mpi_thread_multiple
>>>>>
>>>>> So, apparently the --enable_mpi_thread_multiple option cannot be used by itself, it must be combined with the other three options, otherwise MPI_Init_thread will hang.
>>>>>
>>>>> / Elias
>>>>>
>>>>>
>>>>> Quoting Elias Rudberg<elias.rudberg_at_[hidden]>:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I now tried using --enable-opal-multi-threads (I added an "s" in the end, since "--enable-opal-multi-thread" was not recognized by the configure script).
>>>>>>
>>>>>> However, my test code still hangs in the same way as before.
>>>>>>
>>>>>> To be clear, I now configure like this:
>>>>>> ./configure --enable-mpi-thread-multiple --enable-opal-multi-threads
>>>>>>
>>>>>> but the problem remains: MPI_Init_thread still hangs when trying to run the test case with -np 2.
>>>>>>
>>>>>> Any other ideas?
>>>>>> Are there other configure options that must be used together with --enable-mpi-thread-multiple to make it work?
>>>>>>
>>>>>> Best regards,
>>>>>> Elias
>>>>>>
>>>>>>
>>>>>> Quoting Ralph Castain<rhc_at_[hidden]>:
>>>>>>
>>>>>>> Hmmm...well, your code runs fine for me:
>>>>>>>
>>>>>>> Ralphs-iMac:mpi rhc$ mpirun -n 2 ./thread_init
>>>>>>> Calling MPI_Init_thread...
>>>>>>> Calling MPI_Init_thread...
>>>>>>> MPI_Init_thread returned, provided = 3
>>>>>>> MPI_Init_thread returned, provided = 3
>>>>>>> Ralphs-iMac:mpi rhc$
>>>>>>>
>>>>>>> I think the key, however, is that you also have to configure with --enable-opal-multi-thread.
>>>>>>>
>>>>>>>
>>>>>>> On Jun 16, 2013, at 8:54 AM, Elias Rudberg<elias.rudberg_at_[hidden]> wrote:
>>>>>>>
>>>>>>>> Hello!
>>>>>>>>
>>>>>>>> I would like to report what seems to be a bug in MPI_Init_thread in OpenMPI 1.7.1.
>>>>>>>>
>>>>>>>> The bug can be reproduced with the following test program (test_mpi_thread_support.c):
>>>>>>>> ===========================================
>>>>>>>> #include <mpi.h>
>>>>>>>> #include <stdio.h>
>>>>>>>> int main(int argc, const char* argv[]) {
>>>>>>>> int provided = -1;
>>>>>>>> printf("Calling MPI_Init_thread...\n");
>>>>>>>> MPI_Init_thread(NULL, NULL, MPI_THREAD_MULTIPLE, &provided);
>>>>>>>> printf("MPI_Init_thread returned, provided = %d\n", provided);
>>>>>>>> MPI_Finalize();
>>>>>>>> return 0;
>>>>>>>> }
>>>>>>>> ===========================================
>>>>>>>>
>>>>>>>> When trying to run this when OpenMPI was configured with --enable-mpi-thread-multiple, the program hangs when trying to run using anything more than one process.
>>>>>>>>
>>>>>>>> Steps I use to reproduce this in Ubuntu:
>>>>>>>>
>>>>>>>> (1) Download openmpi-1.7.1.tar.gz
>>>>>>>>
>>>>>>>> (2) Configure like this:
>>>>>>>> ./configure --enable-mpi-thread-multiple
>>>>>>>>
>>>>>>>> (3) make
>>>>>>>>
>>>>>>>> (4) Compile test program like this:
>>>>>>>> mpicc test_mpi_thread_support.c
>>>>>>>>
>>>>>>>> (5) Run like this:
>>>>>>>> mpirun -np 2 ./a.out
>>>>>>>> Then you see the following two lines of output:
>>>>>>>> Calling MPI_Init_thread...
>>>>>>>> Calling MPI_Init_thread...
>>>>>>>> And then it hangs.
>>>>>>>>
>>>>>>>> MPI_Init_thread did not hang in earlier OpenMPI versions (for example it worked in 1.5.* and 1.6.*), so it seems like a bug introduced in 1.7.
>>>>>>>>
>>>>>>>> The description above shows how I reproduce this in Ubuntu on my local desktop computer, but the same problem exists for the OpenMPI 1.7.1 installation at the UPPMAX computer center where I wan to run my code in the end. I don't know all details about how they installed it there, but I know they set --enable-mpi-thread-multiple. So maybe it hangs in 1.7.1 on any computer as long as you use MPI_THREAD_MULTIPLE. At least I have not seen it work anywhere.
>>>>>>>>
>>>>>>>> Do you agree that this is a bug, or am I doing something wrong?
>>>>>>>>
>>>>>>>> Best regards,
>>>>>>>> Elias
>>>>>>>>
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> users mailing list
>>>>>>>> users_at_[hidden]
>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>>>> _______________________________________________
>>>>>>> users mailing list
>>>>>>> users_at_[hidden]
>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>> _______________________________________________
>>>> users mailing list
>>>> users_at_[hidden]
>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>> --
>>>
>>> Dr. Hans Ekkehard Plesser, Associate Professor
>>> Head, Basic Science Section
>>>
>>> Dept. of Mathematical Sciences and Technology
>>> Norwegian University of Life Sciences
>>> PO Box 5003, 1432 Aas, Norway
>>>
>>> Phone +47 6496 5467
>>> Fax +47 6496 5401
>>> Emailhans.ekkehard.plesser_at_[hidden]
>>> Homehttp://arken.umb.no/~plesser
>>>
>>>
>>> _______________________________________________
>>> users mailing list
>>> users_at_[hidden]
>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>
>> _______________________________________________
>> users mailing list
>> users_at_[hidden] <mailto:users_at_[hidden]>
>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>
>
>
> _______________________________________________
> users mailing list
> users_at_[hidden]
> http://www.open-mpi.org/mailman/listinfo.cgi/users
>

-- 
Dipl.-Inform. Paul Kapinos   -   High Performance Computing,
RWTH Aachen University, Center for Computing and Communication
Seffenter Weg 23,  D 52074  Aachen (Germany)
Tel: +49 241/80-24915