I am having two problem with the integration of OpenMPI 1.3 and SGE 6.2u1, which we are new with both. The troubles are getting jobs to suspend/resume and collect cpu time correctly.
For suspend/resume I have added the following to my mpirun command:
--mca orte_forward_job_control 1 --mca plm_rsh_daemonize_qrsh 1
and adjusted the suspend_method for the queue that it’s running in. I have not gotten it to place any process into the T state. Although this is not a huge problem, I hope to have this working in the future.
My main problem is getting the cpu time correct. On a multiple cpu job only the master nodes shows the cpu time correct for that process, the others are very short and not sure what they are measuring. (I believe startup time). Here’s and example:
And from watching the runs that time is close to the wall clock time and match what I see for that single process. Now I have gotten it to give what I believe are correct values, but I have to include --debug-daemons option to our mpirun command. With that I get the following:
I have noticed that when I get the cpu time correctly I get qrsh process that startup (my understanding is that this is what starts the processes on the remote machines) and they stay running until the jobs is finished. When I don’t get the correct cpu time, I see the qrsh processes start on the master node, but die off once they start the process on the remote nodes. The PE environment looks like the following:
Please let me know if I can provide any more information to help figure this out.
Manager, High Performance Computing Facility
Information Sciences - Research Informatics
St. Jude Children’s Research Hospital
332 North Lauderdale
Memphis, TN 38105