Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: Re: [OMPI users] Running OpenMPI on SGI Altix with 4096 cores : very poor performance
From: Jeff Squyres (jsquyres_at_[hidden])
Date: 2011-01-06 17:16:29


On Jan 6, 2011, at 5:07 PM, Gilbert Grosdidier wrote:

> Yes Jeff, I'm pretty sure indeed that hyperthreading is enabled, since 16 CPUs are visible in the /proc/cpuinfo pseudo-file, while it's a 8 core Nehalem node.
>
> However, I always carefully checked that only 8 processes are running on each node. Could it be that they are assigned to 8 hyperthreads but only 4 cores, for example ? Is this actually possible with paffinity set to 1 ?

Yes. I actually had this happen to another user recently; I should add this to the FAQ... (/me adds to to-do list)

Here's what I'm guessing is happening: OMPI's paffinity_alone algorithm is currently pretty stupid. It simply assigns the first MPI process on the node to OS processor ID 0. It then assigned the second MPI process on the node to OS processor ID 1. ...and so on.

However, if hyperthreading is enabled, OS processor ID's 0 and 1 might be 2 hyperthreads on the same core. And therefore OMPI has effectively just bound 2 processes to the same core. Ouch!

The output of lstopo can verify if this is happening: look to see if processor ID's 0 through 7 are on the same 4 cores.

Instead of paffinity_alone, use the mpirun --bind-to-core option; that should bind each MPI process to (the first hyperthread in) its own core.

Sidenote: many improvements are coming to our processor affinity system over the next few releases... See my slides from the Open MPI BOF at SC'10 for some discussion of what's coming:

    http://www.open-mpi.org/papers/sc-2010/

-- 
Jeff Squyres
jsquyres_at_[hidden]
For corporate legal information go to:
http://www.cisco.com/web/about/doing_business/legal/cri/