Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: Re: [OMPI users] busy waiting and oversubscriptions
From: Jeff Squyres (jsquyres) (jsquyres_at_[hidden])
Date: 2014-03-26 06:27:37


On Mar 26, 2014, at 1:31 AM, Andreas Schäfer <gentryx_at_[hidden]> wrote:

>> Even when "idle", MPI processes use all the CPU. I thought I remember
>> someone saying that they will be low priority, and so not pose much of
>> an obstacle to other uses of the CPU.
>
> well, if they're blocking in an MPI call, then they'll be doing a busy
> wait, so each thread will easily churn up 100% CPU load.

+1

>> At any rate, my question is whether, if I have processes that spend most
>> of their time waiting to receive a message, I can put more of them than
>> I have physical cores without much slowdown?
>
> AFAICS there will always be a certain slowdown. Is there a reason why
> you would want to oversubscribe your nodes?

Agreed -- this is not a good idea. It suggests that you should make your existing code more efficient -- perhaps by overlapping communication and computation.

>> Does it make any difference if there's hyperthreading with, e.g., 16
>> virtual CPUs based on 8 physical ones? In general I try to limit to the
>> number of physical cores.
>
> That depends much on the code. If the additional threads run a
> different instruction mix, then you might be able to squeeze out some
> additional performance by adding more than the original 8 threads. But
> I've also seen codes which actually run slower with SMT
> ("hyperthreading").

Also agreed. Hyperthreading is a surprisingly complex issue. Recent generation Intel chips have made hyperthreading more friendly to HPC types of codes, but in my experience, it's still not "enough" to run one MPI process per hyperthread (vs. one per core). But YMMV -- try with your own code and see.

Be aware of a few facts, though:

1. There is a fundamental difference between disabling hyperthreading in the BIOS at power-on time and simply running one MPI process per core. Disabling HT at power-on allocates more hardware resources to the remaining HT that is left is each core (e.g., deeper queues).

2. If you have a single-threaded application, there is little/no benefit to running with hyperthreading enabled.

-- 
Jeff Squyres
jsquyres_at_[hidden]
For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/