Hello Ralph,

I finally solved the problem. My OS had installed another version of openmpi, which I removed. Also, I did not try 'make clean' before 'make'. 

I found the problem by doing a verbose build of my application, and noticed that it was referring to libraries from /usr/lib64/mpi/, which was not correct. This was even when I had set the paths of mpic++ in my make file to /home/Morpheus/CustomInstalledPrograms/OpenMPI-1.4.2/ . I had set the paths of libraries and include files in my makefile to specifically point to my custom MPI installation. So, I am curious as to why open-mpi was trying to refer to libraries in /usr/lib64.

I reinstalled everything fresh, and now it seems to work :)

Thanks for your help.

Best,

Devendra


From: Ralph Castain <rhc@open-mpi.org>
To: devendra rai <rai.devendra@yahoo.co.uk>
Sent: Thursday, 5 January 2012, 20:09
Subject: Re: [OMPI users] Strange Segfault in mpirun: orterun.c:486

Hmm....well, I'm at a bit of a loss. It all certainly looks okay. Can you send the output from configure - should be in config.out? Did you do a "make clean" after your configure and before rebuilding OMPI?


On Jan 5, 2012, at 11:25 AM, devendra rai wrote:

  Hello Ralph,

Here is the command that I use to run my application:

/home/Morpheus/CustomInstalledPrograms/OpenMPI-1.4.2/bin/mpirun -np 2 -v --timestamp-output --tag-output --d kdbg ./MPIDerivedDataTypes

where
MPIDerivedDataTypes is the sample application.

Best,

Devendra


From: Ralph Castain <rhc@open-mpi.org>
To: devendra rai <rai.devendra@yahoo.co.uk>
Sent: Thursday, 5 January 2012, 19:14
Subject: Re: [OMPI users] Strange Segfault in mpirun: orterun.c:486

That is indeed rather strange. The failure occurs at a rather trivial point in the code, well before anything useful can happen. What does your cmd line look like?


On Jan 5, 2012, at 11:07 AM, devendra rai wrote:

   Hello Ralph,

This time, rebuilding with the configure line that I showed did not fix the problem. And, I am mighty clueless (with some desperation thrown in).

Any ideas would be awesome.

Best,

Devendra


From: Ralph Castain <rhc@open-mpi.org>
To: devendra rai <rai.devendra@yahoo.co.uk>; Open MPI Users <users@open-mpi.org>
Sent: Thursday, 5 January 2012, 18:49
Subject: Re: [OMPI users] Strange Segfault in mpirun: orterun.c:486

Guess I'm still confused a bit - are you saying that the rebuild no longer fixes the problem?

Otherwise, I don't see a ready solution. If your system undergoes a significant change, then rebuilding is not an unexpected requirement.


On Jan 5, 2012, at 10:39 AM, devendra rai wrote:

Hello Ralph,

Yes, historically, the rebuild fixed the problem. But, now, how do I rebuild against a new libc? And doesn't mpi build process automatically pick up whatever libc version I have?

I cant seem to google this solution. I already included my configure line for building open-mpi.

Thanks a lot.

Devendra Rai


From: Ralph Castain <rhc@open-mpi.org>
To: devendra rai <rai.devendra@yahoo.co.uk>; Open MPI Users <users@open-mpi.org>
Sent: Thursday, 5 January 2012, 18:32
Subject: Re: [OMPI users] Strange Segfault in mpirun: orterun.c:486

Let me see if I understand correctly. You performed some upgrades on Linux, and then orterun fails. Rebuilding OMPI resolved the problem.

Is that correct? If so, then I'm not terribly surprised - it depends on what got upgraded. For example, if the default libc changed due to a revision upgrade, then you probably do need to rebuild against it.


On Jan 5, 2012, at 10:28 AM, devendra rai wrote:

Hello All,

This has happened to me many times, but a simple rebuild of the Open-MPI generally fixed it.

I had a perfectly working (and tested) MPI application till a few days ago. Since then, there has been some upgrades on my Linux OS (basically, the ones which the system picks up automatically).

Now, I rebuild and rerun the same application, I get this:

mpirun: orterun.c:486: orterun: Assertion `((0xdeafbeedULL << 32) + 0xdeafbeedULL) == ((opal_object_t *) (&cmd_line))->obj_magic_id' failed.
[Morpheus:24572] *** Process received signal ***
[Morpheus:24572] Signal: Aborted (6)
[Morpheus:24572] Signal code:  (-6)
[Morpheus:24572] [ 0] /lib64/libpthread.so.0(+0xf2d0) [0x7fd23c9312d0]
[Morpheus:24572] [ 1] /lib64/libc.so.6(gsignal+0x35) [0x7fd23c5e7ab5]
[Morpheus:24572] [ 2] /lib64/libc.so.6(abort+0x186) [0x7fd23c5e8fb6]
[Morpheus:24572] [ 3] /lib64/libc.so.6(__assert_fail+0xf5) [0x7fd23c5e0385]
[Morpheus:24572] [ 4] /home/Morpheus/CustomInstalledPrograms/OpenMPI-1.4.2/bin/mpirun() [0x4037ed]
[Morpheus:24572] [ 5] /home/Morpheus/CustomInstalledPrograms/OpenMPI-1.4.2/bin/mpirun() [0x4032a4]
[Morpheus:24572] [ 6] /lib64/libc.so.6(__libc_start_main+0xfd) [0x7fd23c5d3bfd]
[Morpheus:24572] [ 7] /home/Morpheus/CustomInstalledPrograms/OpenMPI-1.4.2/bin/mpirun() [0x4031c9]
[Morpheus:24572] *** End of error message ***


I have a custom installation of open-mpi, with the configure script looking like this:

./configure --enable-mpi-threads=yes -prefix=/home/Morpheus/CustomInstalledPrograms/OpenMPI-1.4.2 --enable-mem-profile --enable-mem-debug --enable-debug --enable-trace --enable-memchecker --with-valgrind=/usr --with-threads=posix 

The rebuild of my application was successful.

Where should I fix it? and How?

Thanks for ideas.

Best

Devendra
_______________________________________________
users mailing list
users@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users



_______________________________________________
users mailing list
users@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users