Am 19.02.2008 um 12:49 schrieb Neeraj Chourasia:
I am facing problem while calling mpirun in a loop when using with SGE. My sge version is SGE6.1AR_snapshot3. The script i am submitting via sge is
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
let i=0
while [ $i -lt 100 ]
do
echo "############################################################################################"
echo "Iteration :$i"
/usr/local/openmpi-1.2.4/bin/mpirun -np $NP -hostfile $TMP/machines send
let "i+=1"
echo "############################################################################################"
done
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
Now above script runs well for 15-20 iteration and then fails with following message
in a Tight Integration in SGE the qrsh on the slave node has to perfom some housekeeping, e.g. removing the $TMPDIR (would be an RFE to do this only once atv the end of the job. a) saves time, b) some parallel jobs need persistent $TMPDIR on the slave nodes which must for now be implemented by hand).
Could you put a wait of some seconds after the mpirun before the next iteration? Maybe it will help.
-- Reuti
-------------------------Error Message-------------------------------------------------------------------
error: executing task of job 3869 failed: execution daemon on host "n101" didn't accept task
[n199:11989] ERROR: A daemon on node n101 failed to start as expected.
[n199:11989] ERROR: There may be more information available from
[n199:11989] ERROR: the 'qstat -t' command on the Grid Engine tasks.
[n199:11989] ERROR: If the problem persists, please restart the
[n199:11989] ERROR: Grid Engine PE job
[n199:11989] ERROR: The daemon exited unexpectedly with status 1.
-----------------------------------------------------------------------------------------------------------
When i do ssh to n101, there is no orted and qrsh_starter running. While checking its spool file, i came across following message
-----------------------------------------------Execd spool Error Message---------------------------------
|execd|n101|E|no free queue for job 3869 of user neeraj@n199 (localhost = n101)
-----------------------------------------------------------------------------------------------------------------------
What could be the reason for it.
While checking the mailing list, i come across following link
http://www.open-mpi.org/community/lists/users/2007/03/2771.php
but, i dont think its the same problem. Any help is appreciated.
Regards
Neeraj
