Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: Re: [OMPI users] Fault tolerant ompi - Error: Unable to find a list of active MPIRUN processes on this machine.
From: Hellmüller Roman (hroman_at_[hidden])
Date: 2011-03-31 05:10:31


I noticed that the directory /tmp/openmpi-sessions-hroman_at_cbl1_0 is created on the login nodes but not on the compute nodes. By setting orte_tmpdir_base=/tmp in \$prefix/ect/openmpi-mca-params.conf i could make sure that the session directory is created.

But when i now try to checkpoint an application i'll get:

mpirun noticed that process rank 0 with PID 11175 on node node7 exited on signal 10 (User defined signal 1).

hroman_at_node7 ~/semesterthesis/code/code1_heat1d $ ps -a
 9713 pts/0 00:00:02 pbs_mom
 9761 pts/0 00:00:00 bash
11170 pts/0 00:00:00 mpirun
11175 pts/0 00:00:06 heatft_mpi
11178 pts/1 00:00:00 ps
hroman_at_node7 ~/semesterthesis/code/code1_heat1d $ ompi-checkpoint -v 11170
[node7:11184] [ 0.00 / 0.01] Requested - ...
[node7:11184] [ 0.00 / 0.01] Pending - ...

which never returns. and does not seem to do anything.

do you have an idea what to try or do, to make it work?


Von: users-bounces_at_[hidden] [users-bounces_at_[hidden]]" im Auftrag von "Hellmüller Roman [hroman_at_[hidden]]
Gesendet: Mittwoch, 30. März 2011 16:33
Bis: users_at_[hidden]
Betreff: [OMPI users] Fault tolerant ompi - Error: Unable to find a list of active MPIRUN processes on this machine.


I'm trying to get fault tolerant ompi running on our cluster for my semesterthesis.

On the login node i was successful, checkpointing works.
Since the compute nodes have different kernels, i had to compile blcr on the compute nodes again. blcr on the compute nodes works. after that i installed openmpi (1.5.3) on the compute nodes. Letting a normal mpi program run works. also letting it run with -am ft-enable-cr works, but as soon as i would like to take a checkpoint it crashes:

hroman_at_node15 ~/semesterthesis/code/code1_heat1d $ mpirun -np 4 -am ft-enable-cr ./heatft_mpi

hroman_at_node15 ~ $ ps -a
22488 pts/0 00:00:00 pbs_mom
22536 pts/0 00:00:00 bash
22631 pts/0 00:00:00 mpirun
22633 pts/0 00:00:03 heatft_mpi
22634 pts/0 00:00:03 heatft_mpi
22635 pts/0 00:00:03 heatft_mpi
22636 pts/0 00:00:03 heatft_mpi
22743 pts/1 00:00:00 ps

hroman_at_node15 ~ $ ompi-checkpoint 22631
Error: Unable to find a list of active MPIRUN processes on this machine.
       This could be due to one of the following:
        - The PID specified (22631) is not that of an active MPIRUN.
        - The session directory location could not be found/parsed.

       ompi-checkpoint attempted to find the session directory:
       Check to make sure that this directory exists while the MPIRUN
       process is running.

       Return Code: -13 (Not found)


I've tried it with an other application, that doesn't change anything. I also tried to set the checkpoint directorys in $prefix/ect/openmpi-mca-params.conf but that didn't seem to have any effect. however if i write errors in this file (smth that is no parameter eg. "hello world") it will complain, so it seems to read the file.
I also checked the environement variables but they seem to be ok, as far as i can tell.

do you have an idea where the error could be?

here> (40MB) you'll find the library and the build of openmpi & blcr as well as the env variables and the output of ompi_info. please let me know if more outputs are needed.


users mailing list