Open MPI logo

Hardware Locality Development Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Hardware Locality Development mailing list

Subject: Re: [hwloc-devel] Memory affinity
From: David Singleton (David.Singleton_at_[hidden])
Date: 2011-02-28 16:52:21


On 03/01/2011 08:44 AM, Jeff Squyres wrote:
> On Feb 28, 2011, at 4:39 PM, Brice Goglin wrote:
>
>>> So: binding + pinning = binding (as long as you can ensure that the binding + pinning was atomic!).
>>
>> If the application swaps for real, do you really care about NUMA
>> locality ? It seems to me that the overhead of accessing distant NUMA
>> memory may be negligible against the cost of swapping.
>
> True, but what if the swapping was temporary? The overhead due to swap out/swap in may be tiny in comparison to the longer-lasting effects that it causes (putting your memory far away, when then sucks up memory and bus bandwidth, etc.).
>

Exactly our scenario. We use suspend/resume to run large parallel jobs. There will be a relatively short period of paging (of suspended jobs) while the
incoming job makes room for itself on the NUMA nodes it wants and from then on it has ideal NUMA placement. The killer is when the suspended job is finally
resumed - it's ideal NUMA placement gets partially trashed.