Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: [OMPI users] questions about checkpoint/restart on multiple clusters of MPI
From: fengguang tian (fernyabc_at_[hidden])
Date: 2010-03-22 19:20:43


I set up a cluster of 18 nodes using Open MPI and BLCR library, and the MPI
program runs well on the clusters,
but how to checkpoint the MPI program on this clusters?
for example:
here is what I do for a test:
mpiu_at_nimbus: /mirror$ mpirun -np 50 --hostfile .mpihostfile -am ft-enable-cr
hellompi
 the program will run on the clusters
then ,I enter:
mpiu_at_nimbus: /mirror$ ompi-checkpoint -term $(pidof mpirun)

but the MPI program are not terminated as what happaned on single
machine,although it created a checkpoint file“ompi_global_snapshot_
14030.ckpt“ in the home directory on master node.

then I use ompi-restart command in the master node to restart the MPI
program
mpiu_at_nimbus:/mirror$ ompi-restart ompi_global_snapshot_14030.ckpt

the error information is like:
[nimbus:14545] Error: Unable to access the path
[/home/mpiu/ompi_global_snapshot_14030.ckpt/0/opal_snapshot_29.ckpt]!
--------------------------------------------------------------------------
Error: The filename (opal_snapshot_29.ckpt) is invalid because either you
have not provided a filename
       or provided an invalid filename.
       Please see --help for usage.

--------------------------------------------------------------------------
[nimbus:14609] Error: Unable to access the path
[/home/mpiu/ompi_global_snapshot_14030.ckpt/0/opal_snapshot_34.ckpt]!
--------------------------------------------------------------------------
Error: The filename (opal_snapshot_34.ckpt) is invalid because either you
have not provided a filename
       or provided an invalid filename.
       Please see --help for usage.

--------------------------------------------------------------------------
[nimbus:14685] Error: Unable to access the path
[/home/mpiu/ompi_global_snapshot_14030.ckpt/0/opal_snapshot_39.ckpt]!
--------------------------------------------------------------------------
Error: The filename (opal_snapshot_39.ckpt) is invalid because either you
have not provided a filename
       or provided an invalid filename.
       Please see --help for usage.

--------------------------------------------------------------------------
[nimbus:14737] Error: Unable to access the path
[/home/mpiu/ompi_global_snapshot_14030.ckpt/0/opal_snapshot_44.ckpt]!
--------------------------------------------------------------------------
Error: The filename (opal_snapshot_44.ckpt) is invalid because either you
have not provided a filename
       or provided an invalid filename.
       Please see --help for usage.

--------------------------------------------------------------------------
[nimbus:14798] Error: Unable to access the path
[/home/mpiu/ompi_global_snapshot_14030.ckpt/0/opal_snapshot_49.ckpt]!
--------------------------------------------------------------------------
Error: The filename (opal_snapshot_49.ckpt) is invalid because either you
have not provided a filename
       or provided an invalid filename.
       Please see --help for usage.

--------------------------------------------------------------------------
[nimbus:14317] Error: Unable to access the path
[/home/mpiu/ompi_global_snapshot_14030.ckpt/0/opal_snapshot_4.ckpt]!
--------------------------------------------------------------------------
Error: The filename (opal_snapshot_4.ckpt) is invalid because either you
have not provided a filename
       or provided an invalid filename.
       Please see --help for usage.

--------------------------------------------------------------------------
[nimbus:14331] Error: Unable to access the path
[/home/mpiu/ompi_global_snapshot_14030.ckpt/0/opal_snapshot_9.ckpt]!
--------------------------------------------------------------------------
Error: The filename (opal_snapshot_9.ckpt) is invalid because either you
have not provided a filename
       or provided an invalid filename.
       Please see --help for usage.

--------------------------------------------------------------------------
[nimbus:14381] Error: Unable to access the path
[/home/mpiu/ompi_global_snapshot_14030.ckpt/0/opal_snapshot_14.ckpt]!
--------------------------------------------------------------------------
Error: The filename (opal_snapshot_14.ckpt) is invalid because either you
have not provided a filename
       or provided an invalid filename.
       Please see --help for usage.

--------------------------------------------------------------------------
[nimbus:14408] Error: Unable to access the path
[/home/mpiu/ompi_global_snapshot_14030.ckpt/0/opal_snapshot_19.ckpt]!
--------------------------------------------------------------------------
Error: The filename (opal_snapshot_19.ckpt) is invalid because either you
have not provided a filename
       or provided an invalid filename.
       Please see --help for usage.

--------------------------------------------------------------------------
[nimbus:14483] Error: Unable to access the path
[/home/mpiu/ompi_global_snapshot_14030.ckpt/0/opal_snapshot_24.ckpt]!
--------------------------------------------------------------------------
Error: The filename (opal_snapshot_24.ckpt) is invalid because either you
have not provided a filename
       or provided an invalid filename.
       Please see --help for usage.

--------------------------------------------------------------------------
NO 26
Hello, world, I am 2 of 50 on nimbus

NO 26
Hello, world, I am 12 of 50 on nimbus

NO 26
Hello, world, I am 10 of 50 on nimbus

NO 26
Hello, world, I am 1 of 50 on nimbus

NO 26
Hello, world, I am 8 of 50 on nimbus

NO 26
Hello, world, I am 3 of 50 on nimbus

NO 26
Hello, world, I am 0 of 50 on nimbus

NO 26
Hello, world, I am 5 of 50 on nimbus

NO 26
Hello, world, I am 11 of 50 on nimbus

NO 26
Hello, world, I am 6 of 50 on nimbus

NO 26
Hello, world, I am 17 of 50 on nimbus

NO 26
Hello, world, I am 15 of 50 on nimbus

NO 26
Hello, world, I am 18 of 50 on nimbus

NO 27
Hello, world, I am 2 of 50 on nimbus

NO 26
Hello, world, I am 13 of 50 on nimbus

NO 27
Hello, world, I am 12 of 50 on nimbus

NO 26
Hello, world, I am 7 of 50 on nimbus

NO 27
Hello, world, I am 10 of 50 on nimbus

NO 27
Hello, world, I am 1 of 50 on nimbus

NO 26
Hello, world, I am 21 of 50 on nimbus

NO 27
Hello, world, I am 8 of 50 on nimbus

NO 26
Hello, world, I am 22 of 50 on nimbus

NO 27
Hello, world, I am 3 of 50 on nimbus

NO 26
Hello, world, I am 20 of 50 on nimbus

NO 27
Hello, world, I am 0 of 50 on nimbus

NO 27
Hello, world, I am 5 of 50 on nimbus

NO 26
Hello, world, I am 16 of 50 on nimbus

NO 26
Hello, world, I am 26 of 50 on nimbus

NO 26
Hello, world, I am 23 of 50 on nimbus

NO 26
Hello, world, I am 27 of 50 on nimbus

NO 26
Hello, world, I am 28 of 50 on nimbus

NO 27
Hello, world, I am 11 of 50 on nimbus

NO 27
Hello, world, I am 6 of 50 on nimbus

NO 26
Hello, world, I am 25 of 50 on nimbus

NO 26
Hello, world, I am 31 of 50 on nimbus

NO 27
Hello, world, I am 17 of 50 on nimbus

NO 26
Hello, world, I am 30 of 50 on nimbus

NO 26
Hello, world, I am 43 of 50 on nimbus

NO 27
Hello, world, I am 15 of 50 on nimbus

NO 27
Hello, world, I am 18 of 50 on nimbus

NO 26
Hello, world, I am 33 of 50 on nimbus

NO 26
Hello, world, I am 32 of 50 on nimbus

NO 26
Hello, world, I am 47 of 50 on nimbus

NO 28
Hello, world, I am 2 of 50 on nimbus

NO 26
Hello, world, I am 36 of 50 on nimbus

NO 26
Hello, world, I am 35 of 50 on nimbus

NO 27
Hello, world, I am 13 of 50 on nimbus

NO 26
Hello, world, I am 40 of 50 on nimbus

NO 26
Hello, world, I am 38 of 50 on nimbus

NO 26
Hello, world, I am 37 of 50 on nimbus

NO 28
Hello, world, I am 12 of 50 on nimbus

NO 27
Hello, world, I am 7 of 50 on nimbus

NO 28
Hello, world, I am 10 of 50 on nimbus

NO 26
Hello, world, I am 48 of 50 on nimbus

NO 26
Hello, world, I am 41 of 50 on nimbus

NO 28
Hello, world, I am 1 of 50 on nimbus

NO 26
Hello, world, I am 45 of 50 on nimbus

NO 27
Hello, world, I am 21 of 50 on nimbus

NO 26
Hello, world, I am 42 of 50 on nimbus

NO 26
Hello, world, I am 46 of 50 on nimbus

[nimbus:14312] [[63351,0],0]-[[63351,1],46] mca_oob_tcp_msg_recv: readv
failed: Connection reset by peer (104)
--------------------------------------------------------------------------
mpirun has exited due to process rank 4 with PID 14317 on
node nimbus exiting improperly. There are two reasons this could occur:

1. this process did not call "init" before exiting, but others in
the job did. This can cause a job to hang indefinitely while it waits
for all processes to call "init". By rule, if one process calls "init",
then ALL processes must call "init" prior to termination.

2. this process called "init", but exited without calling "finalize".
By rule, all processes that call "init" MUST call "finalize" prior to
exiting or it will be considered an "abnormal termination"

This may have caused other processes in the application to be
terminated by signals sent by mpirun (as reported here).
--------------------------------------------------------------------------

Cheers
 fengguang