Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: [OMPI users] Using (or not using) Torque/Moab under PBS Pro as the OpenMPI launcher
From: David Whitaker (whitaker_at_[hidden])
Date: 2010-12-17 17:29:03

    At Cray, we have a a Cluster Compatibility Mode (CCM)
   which runs under PBSPro. If we do a qsub,
   we get an allocation of compute nodes and the first
   node in the list of allocated nodes is considered the
   "head node" of the "Cluster".

    The list of allocated nodes is contained in
   the file $PBS_NODEFILE.

     To launch a command on the head node we type:
   ccmrun COMMAND
     In the normal PBSPro environment, we launch
   the OpenMPI mpirun as:
   ccmrun ${OpenMPI_Install}/bin/mpirun <options ..>
     -hostfile host.list -np $NCPUS ./hello_mpi

     This all works great with the "mpirun" command
   launching on the "head node" and the the mpirun
   command using ssh/rsh to launch other processes
   on the other nodes in the virtual cluster. Everything

     On Cray sites, that use PBSPro/Torque/Moab, OpenMPI
   detects that Torque/Moab is present and attempts
   to use Torque/Moab to launch on other nodes. Because of
   the way Cray has Torque/moab configured, this doesn't work.
   This is not a problem with OpenMPI.

     Is there an option to OpenMPI to tell it to ignore
   Torque/Moab and use rsh/ssh and the -hostfile/hosts
   even if Torque/Moab is detected?? I've looked through
   the README and the FAQ but I haven't seen anything.

                      Many thanks,

David Whitaker, Ph.D.                              whitaker_at_[hidden]
Aerospace CFD Specialist                        phone: (651)605-9078
ISV Applications/Cray Inc                         fax: (651)605-9001