Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: Re: [OMPI users] openmpi MPI_Init doesn't work after fork under gdb
From: Geoffrey Irving (irving_at_[hidden])
Date: 2013-02-22 18:46:13


More info: it works fine if I exec after the fork (or equivalently,
run anything via system). This provides a sufficient workaround at
least for me, so it's probably not worth complaining further that a
proprietarily modified decade old version of gdb doesn't work. :)

Thanks,
Geoffrey

On Fri, Feb 22, 2013 at 11:18 AM, George Bosilca <bosilca_at_[hidden]> wrote:
> Geoffrey,
>
> If I understand correctly your problem, I think it is a gdb issue. You should inform gdb that after the fork call instead of following the parent you expect to follow the child.
>
> set follow-fork-mode child (should do the trick).
>
> For more info: http://sourceware.org/gdb/onlinedocs/gdb/Forks.html
>
> George.
>
> On Feb 22, 2013, at 19:55 , Geoffrey Irving <irving_at_[hidden]> wrote:
>
>> No luck with gdb 7.5.1. I can get it to run, but it hits an internal
>> error and bails out. Thus, I'm stuck with Apple's 6.3.50. I suppose
>> I'll try to restructure the app so that I don't have to spawn the mpi
>> process with a fork from the other one.
>>
>> Geoffrey
>>
>> On Thu, Feb 21, 2013 at 4:38 PM, Geoffrey Irving <irving_at_[hidden]> wrote:
>>> No, I don't really have any idea what it's doing. You have to add
>>> manual sleeps and attach another instance of gdb if you want to step
>>> through the child, since I also can't get follow-fork-mode child to
>>> work. I only tried that once, and it segfaulted accessing the zero
>>> pointer destroying the stack in the process. I haven't done manual
>>> stepping or printf searches, so I don't know where in MPI_Init it
>>> breaks.
>>>
>>> I'm trying to install gdb 7.5.1 now (a nontrivial process on Mac,
>>> unfortunately) to see if that fixes it.
>>>
>>> Geoffrey
>>>
>>> On Thu, Feb 21, 2013 at 4:28 PM, Ralph Castain <rhc_at_[hidden]> wrote:
>>>> Hmmm...how about that? Yeah, it doesn't work with the devel trunk either - I'd missed that point.
>>>>
>>>> No idea why, I'm afraid - never tried it before. Are you sure it "crashes"? I'm still getting a child status of "0", but no message output. My guess is that the I/O is being lost for some reason.
>>>>
>>>> On Feb 21, 2013, at 4:09 PM, Geoffrey Irving <irving_at_[hidden]> wrote:
>>>>
>>>>> Actually, I don't see it printing "We're an MPI program!" under gdb,
>>>>> which means it isn't working.
>>>>>
>>>>> Geoffrey
>>>>>
>>>>> On Thu, Feb 21, 2013 at 4:07 PM, Ralph Castain <rhc_at_[hidden]> wrote:
>>>>>> Hmmm...works with 1.6.4 for me on Mac 10.8.2:
>>>>>>
>>>>>> Ralphs-iMac:v1.6 rhc$ ./fork-bug
>>>>>> We're an MPI program!
>>>>>> child status = 0
>>>>>> Ralphs-iMac:v1.6 rhc$ gdb ./fork-bug
>>>>>> GNU gdb 6.3.50-20050815 (Apple version gdb-1820) (Sat Jun 16 02:40:11 UTC 2012)
>>>>>> Copyright 2004 Free Software Foundation, Inc.
>>>>>> GDB is free software, covered by the GNU General Public License, and you are
>>>>>> welcome to change it and/or distribute copies of it under certain conditions.
>>>>>> Type "show copying" to see the conditions.
>>>>>> There is absolutely no warranty for GDB. Type "show warranty" for details.
>>>>>> This GDB was configured as "x86_64-apple-darwin"...Reading symbols for shared libraries ... done
>>>>>>
>>>>>> (gdb) r
>>>>>> Starting program: /Volumes/RHCHD/rhc/openmpi/v1.6/fork-bug
>>>>>> Reading symbols for shared libraries ++............................. done
>>>>>> child status = 0
>>>>>>
>>>>>> Program exited normally.
>>>>>> (gdb)
>>>>>>
>>>>>> Afraid I have no idea why you might be hitting the problem, though...
>>>>>>
>>>>>>
>>>>>> On Feb 21, 2013, at 3:50 PM, Geoffrey Irving <irving_at_[hidden]> wrote:
>>>>>>
>>>>>>> The singleton fork/exec itself is fine, since normal MPI programs work
>>>>>>> under gdb (e.g., fork-bug.c without the fork). gdb is has
>>>>>>> follow-fork-mode set to parent, so it's odd that gdb is looking at the
>>>>>>> child process's trickery at all.
>>>>>>>
>>>>>>> I've confirmed that it's still broken under 1.6.4, unfortunately.
>>>>>>>
>>>>>>> Geoffrey
>>>>>>>
>>>>>>> On Thu, Feb 21, 2013 at 3:36 PM, Ralph Castain <rhc_at_[hidden]> wrote:
>>>>>>>> Singletons fork/exec a daemon to support them - my guess is that gdb may not like it on your machine?
>>>>>>>>
>>>>>>>> FWIW - it runs fine for me using the developer's trunk. You might try with 1.6.4 in case it's a bug in 1.6.0
>>>>>>>>
>>>>>>>>
>>>>>>>> On Feb 21, 2013, at 3:18 PM, Geoffrey Irving <irving_at_[hidden]> wrote:
>>>>>>>>
>>>>>>>>> The attached program illustrates the problem. It forks, and the child
>>>>>>>>> calls MPI_Init. This works fine unless I'm inside gdb. Inside gdb,
>>>>>>>>> MPI_Init silently crashes.
>>>>>>>>>
>>>>>>>>> I'm using OpenMPI 1.6.0 on Mac 10.8.2. I'm running the program
>>>>>>>>> directly, not through mpirun.
>>>>>>>>>
>>>>>>>>> Any ideas what might be wrong?
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Geoffrey
>>>>>>>>>
>>>>>>>>> cone:scratch% /usr/local/bin/mpicc -o fork-bug fork-bug.c
>>>>>>>>> cone:scratch% ./fork-bug
>>>>>>>>> We're an MPI program!
>>>>>>>>> child status = 0
>>>>>>>>> cone:scratch% gdb ./fork-bug
>>>>>>>>> gdb ./fork-bug
>>>>>>>>> GNU gdb 6.3.50-20050815 (Apple version gdb-1824) (Thu Nov 15 10:42:43 UTC 2012)
>>>>>>>>> Copyright 2004 Free Software Foundation, Inc.
>>>>>>>>> GDB is free software, covered by the GNU General Public License, and you are
>>>>>>>>> welcome to change it and/or distribute copies of it under certain conditions.
>>>>>>>>> Type "show copying" to see the conditions.
>>>>>>>>> There is absolutely no warranty for GDB. Type "show warranty" for details.
>>>>>>>>> This GDB was configured as "x86_64-apple-darwin"...Reading symbols for
>>>>>>>>> shared libraries ... done
>>>>>>>>>
>>>>>>>>> (gdb) run
>>>>>>>>> Reading symbols for shared libraries ++............................. done
>>>>>>>>> child status = 5
>>>>>>>>>
>>>>>>>>> Program exited normally.
>>>>>>>>> (gdb) cone:scratch%
>>>>>>>>> <fork-bug.c>_______________________________________________
>>>>>>>>> users mailing list
>>>>>>>>> users_at_[hidden]
>>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>>>>>
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> users mailing list
>>>>>>>> users_at_[hidden]
>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>>>> _______________________________________________
>>>>>>> users mailing list
>>>>>>> users_at_[hidden]
>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> users mailing list
>>>>>> users_at_[hidden]
>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>> _______________________________________________
>>>>> users mailing list
>>>>> users_at_[hidden]
>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>
>>>>
>>>> _______________________________________________
>>>> users mailing list
>>>> users_at_[hidden]
>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>> _______________________________________________
>> users mailing list
>> users_at_[hidden]
>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>
>
> _______________________________________________
> users mailing list
> users_at_[hidden]
> http://www.open-mpi.org/mailman/listinfo.cgi/users