Open MPI logo

Open MPI Development Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Development mailing list

Subject: Re: [OMPI devel] System V Shared Memory for Open MPI: Request for Community Input and Testing
From: N.M. Maclaren (nmm1_at_[hidden])
Date: 2010-05-02 09:48:26


On May 2 2010, Ashley Pittman wrote:
>On 2 May 2010, at 04:03, Samuel K. Gutierrez wrote:
>
> As to performance there should be no difference in use between sys-V
> shared memory and file-backed shared memory, the instructions issued and
> the MMU flags for the page should both be the same so the performance
> should be identical.

Not necessarily, and possibly not so even for far-future Linuces.
On at least one system I used, the poxious kernel wrote the complete
file to disk before returning - all right, it did that for System V
shared memory, too, just to a 'hidden' file! But, if I recall, on
another it did that only for file-backed shared memory - however, it's
a decade ago now and I may be misremembering.

Of course, that's a serious issue mainly for large segments. I was
using multi-GB ones. I don't know how big the ones you need are.

> The one area you do need to keep an eye on for performance is on numa
> machines where it's important which process on a node touches each page
> first, you can end up using different areas (pages, not regions) for
> communicating in different directions between the same pair of processes.
> I don't believe this is any different to mmap backed shared memory
> though.

On some systems it may be, but in bizarre, inconsistent, undocumented
and unpredictable ways :-( Also, there are usually several system (and
sometimes user) configuration options that change the behaviour, so you
have to allow for that. My experience of trying to use those is that
different uses have incompatible requirements, and most of the critical
configuration parameters apply to ALL uses!

In my view, the configuration variability is the number one nightmare
for trying to write portable code that uses any form of shared memory.
ARMCI seem to agree.

>> Because of this, sysv support may be limited to Linux systems - that is,
>> until we can get a better sense of which systems provide the shmctl
>> IPC_RMID behavior that I am relying on.

And, I suggest, whether they have an evil gotcha on one of the areas that
Ashley Pittman noted.

Regards,
Nick Maclaren.