Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: Re: [OMPI users] Strange Segfault in mpirun: orterun.c:486
From: Jeff Squyres (jsquyres_at_[hidden])
Date: 2012-01-05 13:39:41


I thought Ralph answered this one already:

    http://www.open-mpi.org/community/lists/users/2012/01/18102.php

On Jan 5, 2012, at 1:25 PM, devendra rai wrote:

> Hello Ralph,
>
> Here is the command that I use to run my application:
>
> /home/Morpheus/CustomInstalledPrograms/OpenMPI-1.4.2/bin/mpirun -np 2 -v --timestamp-output --tag-output --d kdbg ./MPIDerivedDataTypes
>
> where MPIDerivedDataTypes is the sample application.
>
> Best,
>
> Devendra
>
> From: Ralph Castain <rhc_at_[hidden]>
> To: devendra rai <rai.devendra_at_[hidden]>
> Sent: Thursday, 5 January 2012, 19:14
> Subject: Re: [OMPI users] Strange Segfault in mpirun: orterun.c:486
>
> That is indeed rather strange. The failure occurs at a rather trivial point in the code, well before anything useful can happen. What does your cmd line look like?
>
>
> On Jan 5, 2012, at 11:07 AM, devendra rai wrote:
>
>> Hello Ralph,
>>
>> This time, rebuilding with the configure line that I showed did not fix the problem. And, I am mighty clueless (with some desperation thrown in).
>>
>> Any ideas would be awesome.
>>
>> Best,
>>
>> Devendra
>>
>> From: Ralph Castain <rhc_at_[hidden]>
>> To: devendra rai <rai.devendra_at_[hidden]>; Open MPI Users <users_at_[hidden]>
>> Sent: Thursday, 5 January 2012, 18:49
>> Subject: Re: [OMPI users] Strange Segfault in mpirun: orterun.c:486
>>
>> Guess I'm still confused a bit - are you saying that the rebuild no longer fixes the problem?
>>
>> Otherwise, I don't see a ready solution. If your system undergoes a significant change, then rebuilding is not an unexpected requirement.
>>
>>
>> On Jan 5, 2012, at 10:39 AM, devendra rai wrote:
>>
>>> Hello Ralph,
>>>
>>> Yes, historically, the rebuild fixed the problem. But, now, how do I rebuild against a new libc? And doesn't mpi build process automatically pick up whatever libc version I have?
>>>
>>> I cant seem to google this solution. I already included my configure line for building open-mpi.
>>>
>>> Thanks a lot.
>>>
>>> Devendra Rai
>>>
>>> From: Ralph Castain <rhc_at_[hidden]>
>>> To: devendra rai <rai.devendra_at_[hidden]>; Open MPI Users <users_at_[hidden]>
>>> Sent: Thursday, 5 January 2012, 18:32
>>> Subject: Re: [OMPI users] Strange Segfault in mpirun: orterun.c:486
>>>
>>> Let me see if I understand correctly. You performed some upgrades on Linux, and then orterun fails. Rebuilding OMPI resolved the problem.
>>>
>>> Is that correct? If so, then I'm not terribly surprised - it depends on what got upgraded. For example, if the default libc changed due to a revision upgrade, then you probably do need to rebuild against it.
>>>
>>>
>>> On Jan 5, 2012, at 10:28 AM, devendra rai wrote:
>>>
>>>> Hello All,
>>>>
>>>> This has happened to me many times, but a simple rebuild of the Open-MPI generally fixed it.
>>>>
>>>> I had a perfectly working (and tested) MPI application till a few days ago. Since then, there has been some upgrades on my Linux OS (basically, the ones which the system picks up automatically).
>>>>
>>>> Now, I rebuild and rerun the same application, I get this:
>>>>
>>>> mpirun: orterun.c:486: orterun: Assertion `((0xdeafbeedULL << 32) + 0xdeafbeedULL) == ((opal_object_t *) (&cmd_line))->obj_magic_id' failed.
>>>> [Morpheus:24572] *** Process received signal ***
>>>> [Morpheus:24572] Signal: Aborted (6)
>>>> [Morpheus:24572] Signal code: (-6)
>>>> [Morpheus:24572] [ 0] /lib64/libpthread.so.0(+0xf2d0) [0x7fd23c9312d0]
>>>> [Morpheus:24572] [ 1] /lib64/libc.so.6(gsignal+0x35) [0x7fd23c5e7ab5]
>>>> [Morpheus:24572] [ 2] /lib64/libc.so.6(abort+0x186) [0x7fd23c5e8fb6]
>>>> [Morpheus:24572] [ 3] /lib64/libc.so.6(__assert_fail+0xf5) [0x7fd23c5e0385]
>>>> [Morpheus:24572] [ 4] /home/Morpheus/CustomInstalledPrograms/OpenMPI-1.4.2/bin/mpirun() [0x4037ed]
>>>> [Morpheus:24572] [ 5] /home/Morpheus/CustomInstalledPrograms/OpenMPI-1.4.2/bin/mpirun() [0x4032a4]
>>>> [Morpheus:24572] [ 6] /lib64/libc.so.6(__libc_start_main+0xfd) [0x7fd23c5d3bfd]
>>>> [Morpheus:24572] [ 7] /home/Morpheus/CustomInstalledPrograms/OpenMPI-1.4.2/bin/mpirun() [0x4031c9]
>>>> [Morpheus:24572] *** End of error message ***
>>>>
>>>>
>>>> I have a custom installation of open-mpi, with the configure script looking like this:
>>>>
>>>> ./configure --enable-mpi-threads=yes -prefix=/home/Morpheus/CustomInstalledPrograms/OpenMPI-1.4.2 --enable-mem-profile --enable-mem-debug --enable-debug --enable-trace --enable-memchecker --with-valgrind=/usr --with-threads=posix
>>>>
>>>> The rebuild of my application was successful.
>>>>
>>>> Where should I fix it? and How?
>>>>
>>>> Thanks for ideas.
>>>>
>>>> Best
>>>>
>>>> Devendra
>>>> _______________________________________________
>>>> users mailing list
>>>> users_at_[hidden]
>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>
>>>
>>>
>>> _______________________________________________
>>> users mailing list
>>> users_at_[hidden]
>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>
>>
>>
>
>
>
> _______________________________________________
> users mailing list
> users_at_[hidden]
> http://www.open-mpi.org/mailman/listinfo.cgi/users

-- 
Jeff Squyres
jsquyres_at_[hidden]
For corporate legal information go to:
http://www.cisco.com/web/about/doing_business/legal/cri/