Hi Team,

 

Our users jobs are exiting with below error for random nodes. could you please help us to resolve this issue?

 

[root@bng1grcdc200 output.228472]# cat user_script.stderr

[bng1grcdc181:08381] [[54933,0],0] ORTE_ERROR_LOG: The specified application failed to start in file plm_lsf_module.c at line 308

[bng1grcdc181:08381] lsb_launch failed: 0

--------------------------------------------------------------------------

A daemon (pid unknown) died unexpectedly on signal 1  while attempting to

launch so we are aborting.

 

There may be more information reported by the environment (see above).

 

This may be because the daemon was unable to find all the needed shared

libraries on the remote node. You may set your LD_LIBRARY_PATH to have the

location of the shared libraries on the remote nodes and this will

automatically be forwarded to the remote nodes.

--------------------------------------------------------------------------

--------------------------------------------------------------------------

mpirun noticed that the job aborted, but has no info as to the process

that caused that situation.

--------------------------------------------------------------------------

--------------------------------------------------------------------------

mpirun was unable to cleanly terminate the daemons on the nodes shown

below. Additional manual cleanup may be required - please refer to

the "orte-clean" tool for assistance.

--------------------------------------------------------------------------

        bng1grcdc172 - daemon did not report back when launched

        bng1grcdc154 - daemon did not report back when launched

        bng1grcdc198 - daemon did not report back when launched

        bng1grcdc183 - daemon did not report back when launched

        bng1grcdc187 - daemon did not report back when launched

        bng1grcdc196 - daemon did not report back when launched

        bng1grcdc153 - daemon did not report back when launched

        bng1grcdc173 - daemon did not report back when launched

        bng1grcdc193 - daemon did not report back when launched

        bng1grcdc185 - daemon did not report back when launched

        bng1grcdc176 - daemon did not report back when launched

        bng1grcdc190 - daemon did not report back when launched

        bng1grcdc194 - daemon did not report back when launched

        bng1grcdc156 - daemon did not report back when launched

 

 

Thanks,

Bharati Singh