Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: Re: [OMPI users] Processor/core selection/affinity for large shared memory systems
From: V. Ram (v_r_959_at_[hidden])
Date: 2008-12-04 16:07:27

Ralph H. Castain wrote:

> I confess to confusion. OpenMPI will by default map your processes in
> a round-robin fashion based on process slot. If you are in a resource
> managed environment (e.g., TM or SLURM), then the slots correspond to
> cores. If you are in an unmanaged environment, then your hostfile
> needs to specify a single hostname, and the slots=x number should
> reflect the total number of cores on your machine.

> If you then set mpi_paffinity_alone=1, OMPI will bind each rank to its
> associated core.

> Is that not what you are trying to do?
> Ralph

I probably didn't explain myself well. In this case, the system is not
running a resource manager like SLURM. It is running Linux. If I run
numactl --hardware, then I get:

available: 8 nodes (0-7)
node 0 cpus: 0 1 2 3
node 1 cpus: 4 5 6 7
node 3 cpus: 12 13 14 15
node 4 cpus: 16 17 18 19
node 5 cpus: 20 21 22 23
node 6 cpus: 24 25 26 27
node 7 cpus: 28 29 30 31

where I've elided the memory-related output as well as the node
distances. Just to reiterate, each node here is an AMD processor, part
of the 8-way system; there is no IP networking going on.

What I'd like is, if start a job with mpirun -np 16 <executablename>,
these 16 MPI processes get allocated on continuous "cpus" in numactl
parlance, e.g. cpus 0-15, or 12-27, etc.

As it stands, if I check the cpus allocated to the aforementioned -np 16
job, I see various cores active on multiple sockets, but I don't see
whole sockets (all 4 cores) active at a time on this job.

Does this make more sense?

  V. Ram
-- - A no graphics, no pop-ups email service