Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: [OMPI users] Checkpoint problem
From: Gabriele Fatigati (g.fatigati_at_[hidden])
Date: 2008-08-20 03:16:51


Dear OpenMPI developers,
i'm testing checkpoint and restart with OpenMPI 1.4 nightly. Test machine is
IBM Blade System over Infiniband with 4 processors every communication node.
At the moment, I have some problems. My application is a simply
communication ring between processors, with parametric loop.

*First case:* 8 procs over 2 nodes.

Start command:

$ mpirun -machinefile machinefile -am ft-enable-cr ./ring -t 5000000

The output is:

[node0316:20037] mca: base: components_open: Looking for filem components
[node0316:20037] mca: base: components_open: including only filem
components that are checkpoint enabled
[node0316:20037] mca: base: components_open: (filem) Component rsh is
Checkpointable
[node0316:20037] mca: base: components_open: opening filem components
[node0316:20037] mca: base: components_open: found loaded component rsh
[node0316:20037] mca: base: components_open: component rsh has no
register function
[node0316:20037] filem:rsh: open()
[node0316:20037] filem:rsh: open: priority = 20
[node0316:20037] filem:rsh: open: verbosity = 0
[node0316:20037] filem:rsh: open: cp command = scp
[node0316:20037] filem:rsh: open: rsh command = ssh
[node0316:20037] mca: base: components_open: component rsh open function
successful
[node0316:20037] mca:base:select: Auto-selecting filem components
[node0316:20037] mca:base:select:(filem) Querying component [rsh]
[node0316:20037] mca:base:select:(filem) Query of component [rsh] set
priority to 20
[node0316:20037] mca:base:select:(filem) Selected component [rsh]
[node0316:20037] mca: base: components_open: Looking for snapc components
[node0316:20037] mca: base: components_open: including only snapc
components that are checkpoint enabled
[node0316:20037] mca: base: components_open: (snapc) Component full is
Checkpointable
[node0316:20037] mca: base: components_open: opening snapc components
[node0316:20037] mca: base: components_open: found loaded component full
[node0316:20037] mca: base: components_open: component full has no
register function
[node0316:20037] snapc:full: open()
[node0316:20037] snapc:full: open: priority = 20
[node0316:20037] snapc:full: open: verbosity = 100
[node0316:20037] snapc:full: open: skip_filem = False
[node0316:20037] mca: base: components_open: component full open
function successful
[node0316:20037] mca:base:select: Auto-selecting snapc components
[node0316:20037] mca:base:select:(snapc) Querying component [full]
[node0316:20037] snapc:full: component_query()
[node0316:20037] mca:base:select:(snapc) Query of component [full] set
priority to 20
[node0316:20037] mca:base:select:(snapc) Selected component [full]
[node0316:20037] snapc:full: module_init(1, 1)
[node0316:20037] snapc:full: module_init: Global Snapshot Coordinator
** HANG**

The application doesn't start, and appears locked.

Strace command before mpirun shows the informations below:

poll([{fd=4, events=POLLIN}, {fd=5, events=POLLIN}, {fd=6,
events=POLLIN}], 3, 1000) = 0
poll([{fd=4, events=POLLIN}, {fd=5, events=POLLIN}, {fd=6,
events=POLLIN}], 3, 1000) = 0
poll([{fd=4, events=POLLIN}, {fd=5, events=POLLIN}, {fd=6,
events=POLLIN}], 3, 1000) = 0
poll([{fd=4, events=POLLIN}, {fd=5, events=POLLIN}, {fd=6,
events=POLLIN}], 3, 1000) = 0
poll([{fd=4, events=POLLIN}, {fd=5, events=POLLIN}, {fd=6,
events=POLLIN}], 3, 1000) = 0
poll(...

doing nothing..

*Second case:* 1 node, 4 processor ( without intercommunication over
Infiniband)

In this case, mpirun works well, but the checkpoint procedure fails:

ompi-checkpoint 20109
[node0316:20134] Error: Unable to get the current working directory
[node0316:20134] [[42404,0],0] ORTE_ERROR_LOG: Not found in file
orte-checkpoint.c at line 395
[node0316:20134] HNP with PID 20109 Not found!

I don't understand why OpenMPI doesn't find that log file.

Any idea?

Thanks in advance.

-- 
Gabriele Fatigati
CINECA Systems & Tecnologies Department
Supercomputing Group
Via Magnanelli 6/3, Casalecchio di Reno (BO) Italy
www.cineca.it Tel: +39 051 6171722
g.fatigati_at_[hidden]