We are migrating to openmpi on our large (~1000 node) cluster, and plan
to use it exclusively on a multi-thousand core infiniband cluster in the
near future. We had extensive problems with parallel processes not dying
after a job crash, which was largely solved by switching to the slurm
While orterun supports slurm, it only uses the srun facility to launch
the "orted" daemons, which then start the actual user process
themselves. In our recent migration to openmpi, we have noticed
occasions where orted did not correctly clean up after a parallel job
crash. In most cases the crash was due to an infiniband error. Most
worryingly slurm was not able to cleanup the orted, and it along with
user processes were left running.
At SC07 I was told that there is some talk of using srun to launch both
orted and user processes, or alternatively use srun only. Either would
solve the cleanup problem, in our experience. Is Rolf Castain on this
We use proctrack/linuxproc slurm process tracking plugin. As noted in
the config man page, this may fail to find certain processes and explain
why slurm could not clean up orted effectively.
man slurm.conf(5), version 1.2.22:
NOTE: "proctrack/linuxproc" and "proctrack/pgid" can fail to identify
all processes associated with a job since processes can become a child
of the init process (when the parent process terminates) or change their
process group. To reliably track all processes, one of the other
mechanisms utilizing kernel modifications is preferable.