Open MPI logo

Open MPI Development Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Development mailing list

Subject: Re: [OMPI devel] OpenMPI, PLPA and Linux cpuset/cgroup support
From: Bert Wesarg (bert.wesarg_at_[hidden])
Date: 2009-07-22 12:55:18


On Wed, Jul 22, 2009 at 17:17, Sylvain Jeaugey<sylvain.jeaugey_at_[hidden]> wrote:
> Hi Jeff,
>
> I'm interested in joining the effort, since we will likely have the same
> problem with SLURM's cpuset support.
>
> On Wed, 22 Jul 2009, Jeff Squyres wrote:
>
>> But as to why it's getting EINVAL, that could be wonky.  We might want to
>> take this to the PLPA list and have you run some small, non-MPI examples to
>> ensure that PLPA is parsing your /sys tree properly, etc.
>
> I don't see the /sys implication here. Can you be more precise on which
> files are read to determine placement ?
Most files under /sys/devices/system/cpu/cpu*/topology/*

>
> IIRC, when you are inside a cpuset, you can see all cpus (/sys should be
> unmodified) but calling set_schedaffinity with a mask containing a cpu
> outside the cpuset will return EINVAL.
No. The Linux kernel ands the cpumask of the tasks cpuset with the
given affinity mask. If no cpuset is involved it takes the online
mask. After that, it checks if the resulting mask is a subset of the
online mask and would return -EINVAL.

> The only solution I see to solve this
> would be to get the "allowed" cpus with sched_getaffinity, which should be
> set according to the cpuset mask.
sched_getaffinity() doesn't return the cpuset mask. It returns the
mask the task can run, which is a subset of the cpuset. Also the
initial mask of the task (right after exec) does not to be the cpuset
mask, because the affinity mask is inherited from the parent.

I does not know any C interface to get a tasks cpuset mask (ok,
libcpuset, looks like this lib is debian now, note to myself: check
this). The Cpus_allowed* fields in /proc/<pid>/status are the same as
sched_getaffinity returns and the /proc/<pid>/cpuset needs to be
resolved, i.e. where is the cpuset fs mounted?

Bert
>
> Sylvain
> _______________________________________________
> devel mailing list
> devel_at_[hidden]
> http://www.open-mpi.org/mailman/listinfo.cgi/devel
>