I want to use SGE to run jobs on a cluster with mx and infiniband nodes.
By dividing the nodes into two host groups SGE will submit to either
The interconnect can be specified in the mpirun command with the --mca
btl parameter. However users would then have to decide at runtime which
of the two interconnects should be used, but if the job has to queue
they just want the first available interconnect. I built OpenMPI 1.2.5
with both the mx and ofed options specified. If I understand some of the
FAQs correctly then in that case the --mca parameter is not necessary
and OpenMPI would find the correct interconnect because it is aware of
both mx and ofed libraries. This does not seem to work, the job always
ends up trying to use mx on openib nodes.
Is there a way to inform mpirun, in cooperation with SGE, which btl
should be used?
I would be grateful for any advice