Open MPI logo

Open MPI Development Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Development mailing list

Subject: Re: [OMPI devel] Question about hanging mpirun
From: Ralph Castain (rhc_at_[hidden])
Date: 2011-07-05 14:21:34


On Jul 5, 2011, at 12:11 PM, Colon, Joseanibal wrote:

> Ok I think I figured out what the deadlock in my application was... and please confirm if this makes sense:
> 1. There was an 'if' condition that was met, causing 2 (out of 3) of my processes to call MPI_finalize().
> 2. The remaining process was still trying to run and at some point was requesting calls like MPI_receive(), MPI_send() and MPI_wait() while the other two processes were at MPI_finalize() (althought they would never exit).The application would hang at that point, but the program was too big for me to figure out where exactly the lonely running process would hang.
> 3. I am no expert on openmpi, so I would appreciate it if someone can confirm if this was an expected behavior. I addressed the condition and now all processes run their course.

That is correct behavior for MPI - i.e., if one process is rattling off MPI requests while the others have already entered finalize, then the job will hang since the requests cannot possibly be met and that proc never calls finalize to release completion of the job.

>
> Thanks,
>
> -Joseanibal
>
>
> -----Original Message-----
> From: devel-bounces_at_[hidden] [mailto:devel-bounces_at_[hidden]] On Behalf Of Colon, Joseanibal
> Sent: Friday, July 01, 2011 5:05 PM
> To: Open MPI Developers
> Subject: Re: [OMPI devel] Question about hanging mpirun
>
> Thanks, will do, I'll get back to you soon
>
>
> -----Original Message-----
> From: devel-bounces_at_[hidden] [mailto:devel-bounces_at_[hidden]] On Behalf Of Jeff Squyres
> Sent: Friday, July 01, 2011 5:00 PM
> To: Open MPI Developers
> Subject: Re: [OMPI devel] Question about hanging mpirun
>
> It sounds like you have a deadlock in your MPI application.
>
> You might want to attach a debugger and see where the MPI processes are stuck.
>
>
> On Jul 1, 2011, at 4:49 PM, Ralph Castain wrote:
>
>> I'm afraid there isn't enough info here to advise - I don't know which poll is failing. What function is calling poll?
>>
>> Could be a problem with the event library, but I don't know. Have you tried using "-mca btl sm,self" instead of tcp?
>>
>>
>> On Jul 1, 2011, at 2:37 PM, Colon, Joseanibal wrote:
>>
>>> I got the LD_LIBRARY_PATH correct and I don't have other installations on the target machine, but it doesn't fix it. I had the suspicion about "./configure" building support for stuff on my machine that is not available on the target machine. Unfortunately the machines are not exactly identical, definitely in terms of hardware. The only similarities are the OS and the x86_64 architecture (this is OpenSUSE 11, SP1).
>>> As you correctly guessed I want to run this on a single machine, and all processes are local. There is some intercommunication going on as well, but all using MPI API. I am guessing that my problem has to do with intercommunications (since strace shows infinite calls to 'poll()'), probably because mpirun is trying to use features that were configured on my machine but not present on the target. Does that make sense?
>>> I figured I don't need any fancy support to just run a couple of processes in parallel locally. What would be the most basic configuration I can use to ensure that this will run on my target machine? (a machine that probably doesn't have support for a lot of the components - no IB devices found). I want openmpi to use the simplest form available. Thanks!
>>>
>>> -Joseanibal
>>>
>>>
>>> From: devel-bounces_at_[hidden] [mailto:devel-bounces_at_[hidden]] On Behalf Of Ralph Castain
>>> Sent: Friday, July 01, 2011 3:50 PM
>>> To: Open MPI Developers
>>> Subject: Re: [OMPI devel] Question about hanging mpirun
>>>
>>> Make sure your LD_LIBRARY_PATH will pickup this installation before anything else - it's possible it is picking up an old one.
>>>
>>> I take it that you are running this on a single machine? So all the procs are local?
>>>
>>> Only other issue is that OMPI's configure does a lot of testing to detect the local environment. So you might be building support for things that aren't on your target machine, and vice versa. If you have to do it this way, you need to ensure that the two machines are absolutely identical, both in hardware and software (watch for those installed packages!).
>>>
>>>
>>> On Jul 1, 2011, at 10:42 AM, Colon, Joseanibal wrote:
>>>
>>>
>>> My mpi application is hanging forever when called with mpirun -np >1 (that is 2 or more... not actually typing the '>').
>>>
>>> So I built openmpi 1.4.3 with default options except I used -prefix=/usr/local/openmpi. I compiled an application against it but I need to run this application elsewhere. So brought in my entire installation directory /usr/local/openmpi to this new machine along with my binary to test it. Ran the following command... (If i did't use the -mca options it would print out messages about missing OpenFrabric):
>>> /usr/local/openmpi/bin/mpirun --mca btl tcp,self -np 2 ./my_application
>>>
>>> This actually works for -np 1. But requesting another process makes the call hang forever. 'strace' of the above call shows an never ending calls to "poll" resulting in (timeout) every time.
>>> Executing /usr/local/openmpi/bin/ompi_info still shows the configure and build host as the machine I built on, but I don't know if this may cause a problem. I also see "Thread support: posix (mpi: no, progress: no)"
>>>
>>> Unfortunately I need to do it this way.. I cannot build openmpi on the target machine, so I need to make it portable. This other machine should be the same architecture and OS and everything.
>>>
>>> I should have solved this yesterday, please help, and thanks!
>>>
>>> -Joseanibal
>>> _______________________________________________
>>> devel mailing list
>>> devel_at_[hidden]
>>> http://www.open-mpi.org/mailman/listinfo.cgi/devel
>>>
>>> _______________________________________________
>>> devel mailing list
>>> devel_at_[hidden]
>>> http://www.open-mpi.org/mailman/listinfo.cgi/devel
>>
>> _______________________________________________
>> devel mailing list
>> devel_at_[hidden]
>> http://www.open-mpi.org/mailman/listinfo.cgi/devel
>
>
> --
> Jeff Squyres
> jsquyres_at_[hidden]
> For corporate legal information go to:
> http://www.cisco.com/web/about/doing_business/legal/cri/
>
>
> _______________________________________________
> devel mailing list
> devel_at_[hidden]
> http://www.open-mpi.org/mailman/listinfo.cgi/devel
>
> _______________________________________________
> devel mailing list
> devel_at_[hidden]
> http://www.open-mpi.org/mailman/listinfo.cgi/devel
>
> _______________________________________________
> devel mailing list
> devel_at_[hidden]
> http://www.open-mpi.org/mailman/listinfo.cgi/devel