Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: Re: [OMPI users] Running OpenMPI on SGI Altix with 4096 cores : very poor performance
From: Jeff Squyres (jsquyres_at_[hidden])
Date: 2011-01-07 09:49:41


On Jan 7, 2011, at 5:27 AM, John Hearns wrote:

> Actually, the topic of hyperthreading is interesting, and we should
> discuss it please.
> Hyperthreading is supposedly implemented better and 'properly' on
> Nehalem - I would be interested to see some genuine
> performance measurements with hyperthreading on/off on your machine Gilbert.

FWIW, from what I've seen, and from the recommendations I've heard from Intel, using hyperthreading is still a hit-or-miss proposition with HPC apps. It's true that Nehalem (and later) hyperthreading is much better than it was before. But hyperthreading is still designed to support apps that stall frequently (so the other hyperthread(s) can take over and do useful work while one is stalled). Good HPC apps don't stall much, so hyperthreading still isn't a huge win.

Nehalem (and later) hyperthreading has been discussed on this list at least once or twice before; google through the archives to see if you can dig up the conversations. I have dim recollections of people sending at least some performance numbers...? (I could be wrong here, though)

> Also you don;t need to reboot and change BIOS settings - there was a
> rather niofty technique on this list I think,
> where you disable every second CPU in Linux - which has the same
> effect as switching off hyperthreading.

Yes, you can disable all but one hyperthread on a processor in Linux by:

    # echo 0 > /sys/devices/system/cpu/cpuX/online

where X is an integer from the set listed in hwloc's lstopo output from the P# numbers (i.e., the OS index values, as opposed to the logical index values). Repeat for the 2nd P# value on each core in your machine. You can run lstopo again to verify that they went offline. You can "echo 1" to the same file to bring it back online.

Note that you can't offline X=0.

Note that this technique technically doesn't disable each hyperthread; it just causes Linux to avoid scheduling on it. Disabling hyperthreading in the BIOS is slightly different; you are actually physically disabling all but one thread per core.

The difference is in how resources in a core are split between hyperthreads. When you disable hyperthreading in the BIOS, all the resources in the core are given to the first hyperthread and the 2nd is deactivated (i.e., the OS doesn't even see it at all). When hyperthreading is enabled in the BIOS, the core resources are split between all hyperthreads.

Specifically: causing the OS to simply not schedule on all but the first hyperthread doesn't give those resources back to the first hyperthread; it just effectively ignores all but the first hyperthread.

My understanding is that hyperthreading can only be activated/deactivated at boot time -- once the core resources are allocated to hyperthreads, they can't be changed while running.

Whether disabling the hyperthreads or simply telling Linux not to schedule on them makes a difference performance-wise remains to be seen. I've never had the time to do a little benchmarking to quantify the difference. If someone could rustle up a few cycles (get it?) to test out what the real-world performance difference is between disabling hyperthreading in the BIOS vs. telling Linux to ignore the hyperthreads, that would be awesome. I'd love to see such results.

My personal guess is that the difference is in the noise. But that's a guess.

-- 
Jeff Squyres
jsquyres_at_[hidden]
For corporate legal information go to:
http://www.cisco.com/web/about/doing_business/legal/cri/