Open MPI logo

Open MPI Development Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Development mailing list

Subject: Re: [OMPI devel] System V Shared Memory for Open MPI:Request for Community Input and Testing
From: Samuel K. Gutierrez (samuel_at_[hidden])
Date: 2010-05-03 11:18:12


Hi Jeff,

Sounds like a plan :-).

Thanks!

--
Samuel K. Gutierrez
Los Alamos National Laboratory
On May 3, 2010, at 9:12 AM, Jeff Squyres wrote:
> It might well be that you need a configure test to determine whether  
> this behavior occurs or not.  Heck, it may even need to be a run- 
> time test!  Hrm.
>
> Write a small C program that does something like the following (this  
> is off the top of my head):
>
> fork a child
> child goes to sleep immediately
> sysv alloc a segment
> attach to it
> ipc rm it
> parent wakes up child
> child tries to attach to segment
>
> If that succeeds, then all is good.  If not, then don't use this  
> stuff.
>
>
> On May 3, 2010, at 10:55 AM, Samuel K. Gutierrez wrote:
>
>> Hi all,
>>
>> Does anyone know of a relatively portable solution for querying a
>> given system for the shmctl behavior that I am relying on, or is this
>> going to be a nightmare?  Because, if I am reading this thread
>> correctly, the presence of shmget and Linux is not sufficient for
>> determining an adequate level of sysv support.
>>
>> Thanks!
>>
>> --
>> Samuel K. Gutierrez
>> Los Alamos National Laboratory
>>
>> On May 2, 2010, at 7:48 AM, N.M. Maclaren wrote:
>>
>>> On May 2 2010, Ashley Pittman wrote:
>>>> On 2 May 2010, at 04:03, Samuel K. Gutierrez wrote:
>>>>
>>>> As to performance there should be no difference in use between sys-
>>>> V shared memory and file-backed shared memory, the instructions
>>>> issued and the MMU flags for the page should both be the same so
>>>> the performance should be identical.
>>>
>>> Not necessarily, and possibly not so even for far-future Linuces.
>>> On at least one system I used, the poxious kernel wrote the complete
>>> file to disk before returning - all right, it did that for System V
>>> shared memory, too, just to a 'hidden' file!  But, if I recall, on
>>> another it did that only for file-backed shared memory - however,  
>>> it's
>>> a decade ago now and I may be misremembering.
>>>
>>> Of course, that's a serious issue mainly for large segments.  I was
>>> using multi-GB ones.  I don't know how big the ones you need are.
>>>
>>>> The one area you do need to keep an eye on for performance is on
>>>> numa machines where it's important which process on a node touches
>>>> each page first, you can end up using different areas (pages, not
>>>> regions) for communicating in different directions between the same
>>>> pair of processes. I don't believe this is any different to mmap
>>>> backed shared memory though.
>>>
>>> On some systems it may be, but in bizarre, inconsistent,  
>>> undocumented
>>> and unpredictable ways :-(  Also, there are usually several system
>>> (and
>>> sometimes user) configuration options that change the behaviour, so
>>> you
>>> have to allow for that.  My experience of trying to use those is  
>>> that
>>> different uses have incompatible requirements, and most of the
>>> critical
>>> configuration parameters apply to ALL uses!
>>>
>>> In my view, the configuration variability is the number one  
>>> nightmare
>>> for trying to write portable code that uses any form of shared  
>>> memory.
>>> ARMCI seem to agree.
>>>
>>>>> Because of this, sysv support may be limited to Linux systems -
>>>>> that is,
>>>>> until we can get a better sense of which systems provide the  
>>>>> shmctl
>>>>> IPC_RMID behavior that I am relying on.
>>>
>>> And, I suggest, whether they have an evil gotcha on one of the areas
>>> that
>>> Ashley Pittman noted.
>>>
>>>
>>> Regards,
>>> Nick Maclaren.
>>>
>>>
>>> _______________________________________________
>>> devel mailing list
>>> devel_at_[hidden]
>>> http://www.open-mpi.org/mailman/listinfo.cgi/devel
>>
>> _______________________________________________
>> devel mailing list
>> devel_at_[hidden]
>> http://www.open-mpi.org/mailman/listinfo.cgi/devel
>>
>
>
> -- 
> Jeff Squyres
> jsquyres_at_[hidden]
> For corporate legal information go to:
> http://www.cisco.com/web/about/doing_business/legal/cri/
>
>
> _______________________________________________
> devel mailing list
> devel_at_[hidden]
> http://www.open-mpi.org/mailman/listinfo.cgi/devel