Open MPI logo

Open MPI Development Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Development mailing list

Subject: Re: [OMPI devel] Trunk Commit Heads-up: New Common Shared Memory Component
From: Samuel K. Gutierrez (samuel_at_[hidden])
Date: 2010-08-11 15:15:01


Hi Terry,

On Aug 11, 2010, at 12:34 PM, Terry Dontje wrote:

> I've done some minor testing on Linux looking at resident and shared
> memory sizes for np=4, 8 and 16 jobs. I could not see any
> appreciable differences in sizes in the process between sysv, posix
> or mmap usage in the SM btl.
>
> So I am still somewhat non-plussed about making this the default.
> It seems like the biggest gain out of using posix might be one
> doesn't need to worry about the location of the backing file. This
> seems kind of frivolous to me since there is a warning that happens
> if the backing file is not memory based.

If I'm not mistaken, the warning is only issued if the backing files
is stored on the following file systems: Lustre, NFS, Panasas, and
GPFS (see: opal_path_nfs in opal/util/path.c). Based on the
performance numbers that Sylvain provided on June 9th of this year
(see attached), there was a performance difference between mmap on /
tmp and mmap on a tmpfs-like file system (/dev/shm in that particular
case). Using the new POSIX component provides us with a portable way
to provide similar shared memory performance gains without having to
worry about where the OMPI session directory is rooted.

--
Samuel K. Gutierrez
Los Alamos National Laboratory
>
> I still working on testing the code on Solaris but I don't imagine I  
> will see anything that will change my mind.
>
> --td
>
> Samuel K. Gutierrez wrote:
>>
>> Hi Rich,
>>
>> It's a modification to the existing common sm component.  The  
>> modifications do include the addition of a new POSIX shared memory  
>> facility, however.
>>
>> Sam
>>
>> On Aug 11, 2010, at 10:05 AM, Graham, Richard L. wrote:
>>
>>> Is this a modification of the existing component, or a new  
>>> component ?
>>>
>>> Rich
>>>
>>>
>>> On 8/10/10 10:52 AM, "Samuel K. Gutierrez" <samuel_at_[hidden]> wrote:
>>>
>>> Hi,
>>>
>>> I wanted to give everyone a heads-up about a new POSIX shared memory
>>> component
>>> that has been in the works for a while now and is ready to be pushed
>>> into the
>>> trunk.
>>>
>>> http://bitbucket.org/samuelkgutierrez/ompi_posix_sm_new
>>>
>>> Some highlights:
>>> o New posix component now the new default.
>>>        o May address some of the shared memory performance issues  
>>> users
>>> encounter
>>>           when OMPI's session directories are inadvertently placed  
>>> on a non-
>>> local
>>>           filesystem.
>>> o Silent component failover.
>>>        o In the default case, if the posix component fails  
>>> initialization,
>>>           mmap will be selected.
>>> o The sysv component will only be queried for selection if it is
>>> placed before
>>>    the mmap component (for example, -mca mpi_common_sm
>>> sysv,posix,mmap).  In the
>>>    default case, sysv will never be queried/selected.
>>> o Per some on-list discussion, now unlinking mmaped file in both  
>>> mmap
>>> and posix
>>>    components (see: "System V Shared Memory for Open MPI: Request  
>>> for
>>> Community
>>>    Input and Testing" thread).
>>> o  Assuming local process homogeneity with respect to all utilized
>>> shared
>>>     memory facilities. That is, if one local process deems a
>>> particular shared
>>>     memory facility acceptable, then ALL local processes should be
>>> able to
>>>     utilize that facility. As it stands, this is an important point
>>> because one
>>>     process dictates to all other local processes which common sm
>>> component will
>>>     be selected based on its own, local run-time test.
>>> o Addressed some of George's code reuse concerns.
>>>
>>> If there are no major objections by August 17th, I'll commit the  
>>> code
>>> after the
>>> Tuesday morning conference call.
>>>
>>> Thanks!
>>>
>>> -- 
>>> Samuel K. Gutierrez
>>> Los Alamos National Laboratory
>>>
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> devel mailing list
>>> devel_at_[hidden]
>>> http://www.open-mpi.org/mailman/listinfo.cgi/devel
>>>
>>>
>>> _______________________________________________
>>> devel mailing list
>>> devel_at_[hidden]
>>> http://www.open-mpi.org/mailman/listinfo.cgi/devel
>>
>> _______________________________________________
>> devel mailing list
>> devel_at_[hidden]
>> http://www.open-mpi.org/mailman/listinfo.cgi/devel
>
>
> -- 
> <mime-attachment.gif>
> Terry D. Dontje | Principal Software Engineer
> Developer Tools Engineering | +1.650.633.7054
> Oracle - Performance Technologies
> 95 Network Drive, Burlington, MA 01803
> Email terry.dontje_at_[hidden]
>
> _______________________________________________
> devel mailing list
> devel_at_[hidden]
> http://www.open-mpi.org/mailman/listinfo.cgi/devel



sm-compared.png