Dear OpenMPI developers,
i'm testing checkpoint and restart with OpenMPI 1.4 nightly. Test machine is IBM Blade System over Infiniband with 4 processors every communication node.
At the moment,  I have some problems. My application is  a simply communication ring between processors, with parametric loop.

First case: 8 procs over 2 nodes.

Start command:

$ mpirun -machinefile machinefile -am ft-enable-cr ./ring -t 5000000

The output is:

[node0316:20037] mca: base: components_open: Looking for filem components
[node0316:20037] mca: base: components_open: including only filem
components that are checkpoint enabled
[node0316:20037] mca: base: components_open: (filem) Component rsh is
Checkpointable
[node0316:20037] mca: base: components_open: opening filem components
[node0316:20037] mca: base: components_open: found loaded component rsh
[node0316:20037] mca: base: components_open: component rsh has no
register function
[node0316:20037] filem:rsh: open()
[node0316:20037] filem:rsh: open: priority   = 20
[node0316:20037] filem:rsh: open: verbosity  = 0
[node0316:20037] filem:rsh: open: cp command  = scp
[node0316:20037] filem:rsh: open: rsh command  = ssh
[node0316:20037] mca: base: components_open: component rsh open function
successful
[node0316:20037] mca:base:select: Auto-selecting filem components
[node0316:20037] mca:base:select:(filem) Querying component [rsh]
[node0316:20037] mca:base:select:(filem) Query of component [rsh] set
priority to 20
[node0316:20037] mca:base:select:(filem) Selected component [rsh]
[node0316:20037] mca: base: components_open: Looking for snapc components
[node0316:20037] mca: base: components_open: including only snapc
components that are checkpoint enabled
[node0316:20037] mca: base: components_open: (snapc) Component full is
Checkpointable
[node0316:20037] mca: base: components_open: opening snapc components
[node0316:20037] mca: base: components_open: found loaded component full
[node0316:20037] mca: base: components_open: component full has no
register function
[node0316:20037] snapc:full: open()
[node0316:20037] snapc:full: open: priority    = 20
[node0316:20037] snapc:full: open: verbosity   = 100
[node0316:20037] snapc:full: open: skip_filem  = False
[node0316:20037] mca: base: components_open: component full open
function successful
[node0316:20037] mca:base:select: Auto-selecting snapc components
[node0316:20037] mca:base:select:(snapc) Querying component [full]
[node0316:20037] snapc:full: component_query()
[node0316:20037] mca:base:select:(snapc) Query of component [full] set
priority to 20
[node0316:20037] mca:base:select:(snapc) Selected component [full]
[node0316:20037] snapc:full: module_init(1, 1)
[node0316:20037] snapc:full: module_init: Global Snapshot Coordinator
** HANG**

The application doesn't start, and appears locked.

Strace command before mpirun shows  the informations below:

poll([{fd=4, events=POLLIN}, {fd=5, events=POLLIN}, {fd=6,
events=POLLIN}], 3, 1000) = 0
poll([{fd=4, events=POLLIN}, {fd=5, events=POLLIN}, {fd=6,
events=POLLIN}], 3, 1000) = 0
poll([{fd=4, events=POLLIN}, {fd=5, events=POLLIN}, {fd=6,
events=POLLIN}], 3, 1000) = 0
poll([{fd=4, events=POLLIN}, {fd=5, events=POLLIN}, {fd=6,
events=POLLIN}], 3, 1000) = 0
poll([{fd=4, events=POLLIN}, {fd=5, events=POLLIN}, {fd=6,
events=POLLIN}], 3, 1000) = 0
poll(...

doing nothing..

Second case: 1 node, 4 processor ( without intercommunication over Infiniband)

In this case, mpirun works well, but the checkpoint procedure fails:

ompi-checkpoint 20109
[node0316:20134] Error: Unable to get the current working directory
[node0316:20134] [[42404,0],0] ORTE_ERROR_LOG: Not found in file
orte-checkpoint.c at line 395
[node0316:20134] HNP with PID 20109 Not found!

I don't understand why OpenMPI doesn't find that log file.

Any idea?

Thanks in advance.





--
Gabriele Fatigati

CINECA Systems & Tecnologies Department

Supercomputing Group

Via Magnanelli 6/3, Casalecchio di Reno (BO) Italy

www.cineca.it Tel: +39 051 6171722

g.fatigati@cineca.it