Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: [OMPI users] segfault when resuming on different host
From: Lloyd Brown (lloyd_brown_at_[hidden])
Date: 2011-12-29 13:19:47


Hi, all.

I'm in the middle of testing some of the checkpoint/restart capabilities
of OpenMPI with BLCR on our cluster. I've been able to checkpoint and
restart successfully when I restart on the same nodes as it was running
previously. But when I try to restart on a different host, I always get
an error like this:

> $ ompi-restart ompi_global_snapshot_15935.ckpt
> --------------------------------------------------------------------------
> mpirun noticed that process rank 1 with PID 15201 on node m5stage-1-2.local exited on signal 11 (Segmentation fault).
> --------------------------------------------------------------------------

Now, it's very possible that I've missed something during the setup, or
that despite my failure to find it while searching the mailing list,
that this is already answered somewhere, but none of the threads I could
find seemed to apply (eg. cr_restart *is* installed, etc.).

I'm attaching a tarball that contains the source code of the very-simple
test application, as well as some example output of "ompi_info --all"
and "ompi_info -v ompi full --parsable". I don't know if this will be
useful or not.

This is being tested on CentOS v5.4 with BLCR v0.8.4. I've seen this
problem with OpenMPI v1.4.2, v1.4.4, and v1.5.4.

If anyone has any ideas on what's going on, or how to best debug this,
I'd love to hear about it.

I don't mind doing the legwork too, but I'm just stumped where to go
from here. I have some core files, but I'm having trouble getting the
symbols from the backtrace in gdb. Maybe I'm doing it wrong.

TIA,

-- 
Lloyd Brown
Systems Administrator
Fulton Supercomputing Lab
Brigham Young University
http://marylou.byu.edu