Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: Re: [OMPI users] ompi-checkpoint problem on shared storage
From: Dave Schulz (dschulz_at_[hidden])
Date: 2011-09-27 17:22:56

Thanks Josh,

Just yesterday I stumbled upon another interesting detail about this
issue. While reconfiguring things, I accidentally ran as root, and the
checkpointing all succeeded. I'm not sure though how to go about
finding what file things are hanging up on. I've compared straces as
root and user and can't see anything out of the ordinary that root can
read or if necessary write to. My /tmp is 777 sticky, my checkpoint
directory (currently ~) is writable by the user. The only files that
root can read but users can't that appear in the strace are these files:

open("/sys/devices/system/cpu/cpu1/online", O_RDONLY) = 3

Root receives a file descriptor as shown above, users get this:

open("/sys/devices/system/cpu/cpu2/online", O_RDONLY) = -1 EACCES
(Permission denied)

Which is what would be expected as the perms on the "online" files are
600. I don't know if this is actually a problem or not. I did try on a
"sacrificial" node to change the perms to 644, but it didn't help. So I
don't really think this is related to the cause. I'm simply listing the
difference that I found between the open syscalls when running as root
vs. running as a user.

Finally, I tried installing a vanilla CentOS 5.6 machine on the same
hardware, Installing openmpi 1.4.3, blcr 0.8.3, Mellanox OFED 1.5.3 and
using a vanilla NFS server on which I see the same symptoms.

So, contrary to what was previously thought, this really isn't caused by
the particular type of network filesystem. Still, things work perfectly
to /tmp (for the checkpoint directory). I can still read the
application and input from the NFS server.

I also noted that when I run with lots of verbosity, I get messages like

Snapshot Ref.: 0 ompi_global_snapshot_9333.ckpt
mpirun noticed that process rank 0 with PID 9338 on node b15 exited on
signal 0 (Unknown signal 0).
2 total processes killed (some possibly by mpirun during cleanup)
[b15:09333] sess_dir_finalize: job session dir not empty - leaving
[b15:09333] snapc:full: module_finalize()
[b15:09333] snapc:full: close()
[b15:09333] mca: base: close: component full closed
[b15:09333] mca: base: close: unloading component full
[b15:09333] filem:rsh: close()
[b15:09333] mca: base: close: component rsh closed
[b15:09333] mca: base: close: unloading component rsh
[b15:09333] sess_dir_finalize: proc session dir not empty - leaving
orterun: exiting with status 0

Specifically the sess_dir_finalize line. It doesn't seem to be cleaning
itself up. it leaves these in the /tmp directory for every run (even
successfully checkpointed and terminated runs):

prw-r----- 1 root root 0 Sep 27 15:21 opal_cr_prog_read.9584
prw-r----- 1 root root 0 Sep 27 15:21 opal_cr_prog_read.9585
prw-r----- 1 root root 0 Sep 27 15:21 opal_cr_prog_write.9584
prw-r----- 1 root root 0 Sep 27 15:21 opal_cr_prog_write.9585

I believe those are pipes. But why they aren't cleaned up after the
checkpoint completes, I don't understand as the job may be restarted on
a different batch of nodes, so these will start accumulating over time.

Any ideas?


On 09/23/2011 01:24 PM, Josh Hursey wrote:
> It sounds like there is a race happening in the shutdown of the
> processes. I wonder if the app is shutting down in a way that mpirun
> does not quite like.
> I have not tested the C/R functionality in the 1.4 series in a long
> time. Can you give it a try with the 1.5 series, and see if there is
> any variation? You might also try the trunk, but I have not tested it
> recently enough to know if things are still working correctly or not
> (have others?).
> I'll file a ticket so we do not lose track of the bug. Hopefully we
> will get to it soon.
> Thanks,
> Josh
> On Fri, Sep 23, 2011 at 3:08 PM, Dave Schulz<dschulz_at_[hidden]> wrote:
>> Hi Everyone.
>> I've been trying to figure out an issue with ompi-checkpoint/blcr. The
>> symptoms seem to be related to what filesystem the
>> snapc_base_global_snapshot_dir is located on.
>> I wrote a simple mpi program where rank 0 sends to 1, 1 to 2, etc. then the
>> highest sends to 0. then it waits 1 sec and repeats.
>> I'm using openmpi-1.4.3 and when I run "ompi-checkpoint --term
>> <pidofmpirun>" on the shared filesystems, the ompi-checkpoint returns a
>> checkpoint reference, the worker processes go away, but the mpirun remains
>> but is stuck (It dies right away if I run kill on it -- so it's responding
>> to SIGTERM). If I attach an strace to the mpirun, I get the following from
>> strace forever:
>> poll([{fd=4, events=POLLIN}, {fd=5, events=POLLIN}, {fd=6, events=POLLIN},
>> {fd=8, events=POLLIN}, {fd=10, events=POLLIN}, {fd=0, events=POLLIN}], 6,
>> 1000) = 0 (Timeout)
>> poll([{fd=4, events=POLLIN}, {fd=5, events=POLLIN}, {fd=6, events=POLLIN},
>> {fd=8, events=POLLIN}, {fd=10, events=POLLIN}, {fd=0, events=POLLIN}], 6,
>> 1000) = 0 (Timeout)
>> poll([{fd=4, events=POLLIN}, {fd=5, events=POLLIN}, {fd=6, events=POLLIN},
>> {fd=8, events=POLLIN}, {fd=10, events=POLLIN}, {fd=0, events=POLLIN}], 6,
>> 1000) = 0 (Timeout)
>> I'm running with:
>> mpirun -machinefile machines -am ft-enable-cr ./mpiloop
>> the "machines" file simply has the local hostname listed a few times. I've
>> tried 2 and 8. I can try up to 24 as this node is a pretty big one if it's
>> deemed useful. Also, there's 256Gb of RAM. And it's Opteron 6 core, 4
>> socket if that helps.
>> I initially installed this on a test system with only local harddisks and
>> standard nfs on Centos 5.6 where everything worked as expected. When I
>> moved over to the production system things started breaking. The filesystem
>> is the major software difference. The shared filesystems are Ibrix and that
>> is where the above symptoms started to appear.
>> I haven't even moved on to multi-node mpi runs as I can't even get this to
>> work for any number of processes on the local machine except if I set the
>> checkpoint directory to /tmp which is on a local xfs harddisk. If I put the
>> checkpoints on any shared directory, things fail.
>> I've tried a number of *_verbose mca parameters and none of them seem to
>> issue any messages at the point of checkpoint, only when I give-up and send
>> kill `pidof mpirun` are there any further messages.
>> openmpi is compiled with:
>> ./configure --prefix=/global/software/openmpi-blcr
>> --with-blcr=/global/software/blcr
>> --with-blcr-libdir=/global/software/blcr/lib/ --with-ft=cr
>> --enable-ft-thread --enable-mpi-threads --with-openib --with-tm
>> and blcr only has a prefix to put it in /global/software/blcr otherwise it's
>> vanilla. Both are compiled with the default gcc.
>> One final note, is that occasionally it does succeed and terminate. But it
>> seems completely random.
>> What I'm wondering is has anyone else seen symptoms like this -- especially
>> where the mpirun doesn't quit after a checkpoint with --term but the worker
>> processes do?
>> Also, is there some sort of somewhat unusual filesystem semantic that our
>> shared filesystem may not support that ompi/ompi-checkpoint is needing?
>> Thanks for any insights you may have.
>> -Dave
>> _______________________________________________
>> users mailing list
>> users_at_[hidden]

Dave Schulz
Research Computing Services
Information Technologies
University of Calgary
(403) 220-2102