Interesting. No, we don't take PLPA cpu sets into account when
retrieving the allocation.
Just to be clear: from an OMPI perspective, I don't think this is an
issue of binding, but rather an issue of allocation. If we knew we had
been allocated only a certain number of cores on a node, then we would
only map that many procs to the node. When we subsequently "bind", we
should then bind those procs to the correct cores (I think).
Could you check this? You can run a trivial job using the -npernode x
option, where x matched the #cores you were allocated on the nodes.
If you do this, do we bind to the correct cores?
If we do, then that would confirm that we just aren't picking up the
right number of cores allocated to us. If it is wrong, then this is a
PLPA issue where it isn't binding to the right core.
On Jul 15, 2009, at 12:28 AM, Chris Samuel wrote:
> Hi all,
> Not sure if this is a OpenMPI query or a PLPA query,
> but given that PLPA seems to have some support for it
> already I thought I'd start here. :-)
> We run a quad core Opteron cluster with Torque 2.3.x
> which uses the kernels cpuset support to constrain
> a job to just the cores it has been allocated.
> However, we are seeing occasionally that where a job
> has been allocated multiple cores on the same node we
> get two compute bound MPI processes in the job scheduled
> onto the same core (obviously a kernel issue).
> So CPU affinity would be an obvious solution, but it
> needs to be done with reference to the cores that are
> available to it in its cpuset.
> This information is already retrievable by PLPA (for
> instance "plpa-taskset -cp $$" will retrieve the cores
> allocated to the shell you run the command from) but
> I'm not sure if OpenMPI makes use of this when binding
> CPUs using the linux paffinity MCA parameter ?
> Our testing (with 1.3.2) seems to show it doesn't, and
> I don't think there are any significant differences with
> the snapshots in 1.4.
> Am I correct in this ? If so, are there any plans to
> make it do this ?
> Christopher Samuel - (03) 9925 4751 - Systems Manager
> The Victorian Partnership for Advanced Computing
> P.O. Box 201, Carlton South, VIC 3053, Australia
> VPAC is a not-for-profit Registered Research Agency
> devel mailing list