Open MPI logo

Open MPI Development Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Development mailing list

Subject: Re: [OMPI devel] Trunk Commit Heads-up: New Common Shared Memory Component
From: Samuel K. Gutierrez (samuel_at_[hidden])
Date: 2010-08-23 12:07:52


Code is in (see r23633). Note: mmap is still the default.

--
Samuel K. Gutierrez
Los Alamos National Laboratory
On Aug 12, 2010, at 11:37 AM, Samuel K. Gutierrez wrote:
> Sorry, I should have included the link containing the discussion of  
> the plot.
>
> http://www.open-mpi.org/community/lists/devel/2010/06/8078.php
>
> --
> Samuel K. Gutierrez
> Los Alamos National Laboratory
>
> On Aug 12, 2010, at 11:20 AM, Terry Dontje wrote:
>
>> Sorry Rich, I didn't realize there was a graph attached at the end  
>> of message.  In other words my comments are not applicable because  
>> I really didn't know you were asking about the graph.  I agree it  
>> would be nice to know what the graph was plotting.
>>
>> --td
>> Terry Dontje wrote:
>>>
>>> Graham, Richard L. wrote:
>>>>
>>>> Stupid question:
>>>>    What is being plotted, and what are the units ?
>>>>
>>>> Rich
>>>>
>>> MB of Resident and Shared memory as gotten from top (on linux).   
>>> The values for each of the processes run cases seem to be the same  
>>> between posix, mmap and sysv.
>>>
>>> --td
>>>> On 8/11/10 3:15 PM, "Samuel K. Gutierrez" <samuel_at_[hidden]> wrote:
>>>>
>>>> Hi Terry,
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On Aug 11, 2010, at 12:34 PM, Terry Dontje wrote:
>>>>
>>>>
>>>>  I've done some minor testing on Linux looking at resident and  
>>>> shared memory sizes for np=4, 8 and 16 jobs.  I could not see any  
>>>> appreciable differences in sizes in the process between sysv,  
>>>> posix or mmap usage in the SM btl.
>>>>
>>>>  So I am still somewhat non-plussed about making this the  
>>>> default.  It seems like the biggest gain out of using posix might  
>>>> be one doesn't need to worry about the location of the backing  
>>>> file.  This seems kind of frivolous to me since there is a  
>>>> warning that happens if the backing file is not memory based.
>>>>
>>>> If I'm not mistaken, the warning is only issued if the backing  
>>>> files is stored on the following file systems: Lustre, NFS,  
>>>> Panasas, and GPFS  (see: opal_path_nfs in opal/util/path.c).   
>>>> Based on the performance numbers that Sylvain provided on June  
>>>> 9th of this year (see attached),  there was a performance  
>>>> difference between mmap on /tmp and mmap on a tmpfs-like file  
>>>> system (/dev/shm in that particular case).  Using the new POSIX  
>>>> component provides us with a portable way to provide similar  
>>>> shared memory performance gains without having to worry about  
>>>> where the OMPI session directory is rooted.
>>>>
>>>> --
>>>> Samuel K. Gutierrez
>>>> Los Alamos National Laboratory
>>>>
>>>> [cid:3364459484_11867134]
>>>>
>>>>
>>>>  I still working on testing the code on Solaris but I don't  
>>>> imagine I will see anything that will change my mind.
>>>>
>>>>  --td
>>>>
>>>>  Samuel K. Gutierrez wrote:
>>>> Hi Rich,
>>>>
>>>>  It's a modification to the existing common sm component.  The  
>>>> modifications do include the addition of a new POSIX shared  
>>>> memory facility, however.
>>>>
>>>>  Sam
>>>>
>>>>  On Aug 11, 2010, at 10:05 AM, Graham, Richard L. wrote:
>>>>
>>>>
>>>> Is this a modification of the existing component, or a new  
>>>> component ?
>>>>
>>>>  Rich
>>>>
>>>>
>>>>  On 8/10/10 10:52 AM, "Samuel K. Gutierrez" <samuel_at_[hidden]> <mailto:samuel_at_[hidden] 
>>>> >  wrote:
>>>>
>>>>  Hi,
>>>>
>>>>  I wanted to give everyone a heads-up about a new POSIX shared  
>>>> memory
>>>>  component
>>>>  that has been in the works for a while now and is ready to be  
>>>> pushed
>>>>  into the
>>>>  trunk.
>>>>
>>>>  http://bitbucket.org/samuelkgutierrez/ompi_posix_sm_new
>>>>
>>>>  Some highlights:
>>>>  o New posix component now the new default.
>>>>         o May address some of the shared memory performance  
>>>> issues users
>>>>  encounter
>>>>            when OMPI's session directories are inadvertently  
>>>> placed on a non-
>>>>  local
>>>>            filesystem.
>>>>  o Silent component failover.
>>>>         o In the default case, if the posix component fails  
>>>> initialization,
>>>>            mmap will be selected.
>>>>  o The sysv component will only be queried for selection if it is
>>>>  placed before
>>>>     the mmap component (for example, -mca mpi_common_sm
>>>>  sysv,posix,mmap).  In the
>>>>     default case, sysv will never be queried/selected.
>>>>  o Per some on-list discussion, now unlinking mmaped file in both  
>>>> mmap
>>>>  and posix
>>>>     components (see: "System V Shared Memory for Open MPI:  
>>>> Request for
>>>>  Community
>>>>     Input and Testing" thread).
>>>>  o  Assuming local process homogeneity with respect to all utilized
>>>>  shared
>>>>      memory facilities. That is, if one local process deems a
>>>>  particular shared
>>>>      memory facility acceptable, then ALL local processes should be
>>>>  able to
>>>>      utilize that facility. As it stands, this is an important  
>>>> point
>>>>  because one
>>>>      process dictates to all other local processes which common sm
>>>>  component will
>>>>      be selected based on its own, local run-time test.
>>>>  o Addressed some of George's code reuse concerns.
>>>>
>>>>  If there are no major objections by August 17th, I'll commit the  
>>>> code
>>>>  after the
>>>>  Tuesday morning conference call.
>>>>
>>>>  Thanks!
>>>>
>>>>  --
>>>>  Samuel K. Gutierrez
>>>>  Los Alamos National Laboratory
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>  _______________________________________________
>>>>  devel mailing list
>>>>  devel_at_[hidden]
>>>>  http://www.open-mpi.org/mailman/listinfo.cgi/devel
>>>>
>>>>
>>>>  _______________________________________________
>>>>  devel mailing list
>>>>  devel_at_[hidden]
>>>>  http://www.open-mpi.org/mailman/listinfo.cgi/devel
>>>>
>>>>
>>>>  _______________________________________________
>>>>  devel mailing list
>>>>  devel_at_[hidden]
>>>>  http://www.open-mpi.org/mailman/listinfo.cgi/devel
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> devel mailing list
>>>> devel_at_[hidden]
>>>> http://www.open-mpi.org/mailman/listinfo.cgi/devel
>>>
>>>
>>> -- 
>>> <mime-attachment.gif>
>>> Terry D. Dontje | Principal Software Engineer
>>> Developer Tools Engineering | +1.650.633.7054
>>> Oracle - Performance Technologies
>>> 95 Network Drive, Burlington, MA 01803
>>> Email terry.dontje_at_[hidden]
>>>
>>
>>
>> -- 
>> <mime-attachment.gif>
>> Terry D. Dontje | Principal Software Engineer
>> Developer Tools Engineering | +1.650.633.7054
>> Oracle - Performance Technologies
>> 95 Network Drive, Burlington, MA 01803
>> Email terry.dontje_at_[hidden]
>>
>> _______________________________________________
>> devel mailing list
>> devel_at_[hidden]
>> http://www.open-mpi.org/mailman/listinfo.cgi/devel
>
> _______________________________________________
> devel mailing list
> devel_at_[hidden]
> http://www.open-mpi.org/mailman/listinfo.cgi/devel