The question was raised on this list a short while ago about potentially
incorrect behavior by ORTE/OMPI in response to SIGUSR2 being sent to
application procs. I have spent some time chasing this down, and it does
-not- appear to be an issue within our systems.
What I have found is that if you send a SIGUSR1/2 to mpirun, mpirun and the
daemons correctly transmit the provided signal to the application processes.
Neither mpirun nor the daemons directly respond to it themselves.
If the application process has defined its own signal handler to trap
USR1/2, then the application process will successfully do so. Everything
seems to work fine - the daemon does -not- get a callback nor in any way
take action to the fact that the proc received this signal - unless the
process' signal handler orders the process to exit! In this case, the
environment reports to the orted that the process exit'd during a signal
handler, which results in a terminated-by-signal status.
You can, of course, get around this by simply not exiting from within the
signal handler. Instead, set a flag and return from the handler, then have
an appropriate routine check the flag and exit. I have done that in several
codes and would be happy to advise you on how to do it. With this technique,
you clear the signal and the environment will not report you as
However, if the application process has -not- defined its own signal
handler, some native environments terminate the process when it receives
SIGUSR1/2! This occurred for me under SLURM on the odin cluster, and under
TM on our RRZ cluster. I cannot say it is a universal situation and would
welcome more feedback from people with access to other environments.
This termination is dutifully reported to the orted, which notes that the
proc was terminated-by-signal. The orted does not check to see -which-
signal was used to terminate the proc.
By our own design requirements, the response to a termination-by-signal of a
process is to abort the job. If we want to modify that, it would be simple
to say "except if it was a SIGUSR1/2 signal". I have no issue with making
that change, but please note that it -is- a change in our defined behavior,
and a change from what has been our behavior since the beginning of the
Let me know if you want to change the design requirement and we can take
care of it.