Open MPI logo

Open MPI Development Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Development mailing list

Subject: [OMPI devel] OpenMPI, PLPA and Linux cpuset/cgroup support
From: Chris Samuel (csamuel_at_[hidden])
Date: 2009-07-15 02:28:11

Hi all,

Not sure if this is a OpenMPI query or a PLPA query,
but given that PLPA seems to have some support for it
already I thought I'd start here. :-)

We run a quad core Opteron cluster with Torque 2.3.x
which uses the kernels cpuset support to constrain
a job to just the cores it has been allocated.

However, we are seeing occasionally that where a job
has been allocated multiple cores on the same node we
get two compute bound MPI processes in the job scheduled
onto the same core (obviously a kernel issue).

So CPU affinity would be an obvious solution, but it
needs to be done with reference to the cores that are
available to it in its cpuset.

This information is already retrievable by PLPA (for
instance "plpa-taskset -cp $$" will retrieve the cores
allocated to the shell you run the command from) but
I'm not sure if OpenMPI makes use of this when binding
CPUs using the linux paffinity MCA parameter ?

Our testing (with 1.3.2) seems to show it doesn't, and
I don't think there are any significant differences with
the snapshots in 1.4.

Am I correct in this ? If so, are there any plans to
make it do this ?


Christopher Samuel - (03) 9925 4751 - Systems Manager
 The Victorian Partnership for Advanced Computing
 P.O. Box 201, Carlton South, VIC 3053, Australia
VPAC is a not-for-profit Registered Research Agency