Open MPI logo

Open MPI Development Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Development mailing list

Subject: Re: [OMPI devel] Trunk Commit Heads-up: New Common Shared Memory Component
From: Samuel K. Gutierrez (samuel_at_[hidden])
Date: 2010-08-12 13:37:03


Sorry, I should have included the link containing the discussion of
the plot.

http://www.open-mpi.org/community/lists/devel/2010/06/8078.php

--
Samuel K. Gutierrez
Los Alamos National Laboratory
On Aug 12, 2010, at 11:20 AM, Terry Dontje wrote:
> Sorry Rich, I didn't realize there was a graph attached at the end  
> of message.  In other words my comments are not applicable because I  
> really didn't know you were asking about the graph.  I agree it  
> would be nice to know what the graph was plotting.
>
> --td
> Terry Dontje wrote:
>>
>> Graham, Richard L. wrote:
>>>
>>> Stupid question:
>>>    What is being plotted, and what are the units ?
>>>
>>> Rich
>>>
>> MB of Resident and Shared memory as gotten from top (on linux).   
>> The values for each of the processes run cases seem to be the same  
>> between posix, mmap and sysv.
>>
>> --td
>>> On 8/11/10 3:15 PM, "Samuel K. Gutierrez" <samuel_at_[hidden]> wrote:
>>>
>>> Hi Terry,
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> On Aug 11, 2010, at 12:34 PM, Terry Dontje wrote:
>>>
>>>
>>>  I've done some minor testing on Linux looking at resident and  
>>> shared memory sizes for np=4, 8 and 16 jobs.  I could not see any  
>>> appreciable differences in sizes in the process between sysv,  
>>> posix or mmap usage in the SM btl.
>>>
>>>  So I am still somewhat non-plussed about making this the  
>>> default.  It seems like the biggest gain out of using posix might  
>>> be one doesn't need to worry about the location of the backing  
>>> file.  This seems kind of frivolous to me since there is a warning  
>>> that happens if the backing file is not memory based.
>>>
>>> If I'm not mistaken, the warning is only issued if the backing  
>>> files is stored on the following file systems: Lustre, NFS,  
>>> Panasas, and GPFS  (see: opal_path_nfs in opal/util/path.c).   
>>> Based on the performance numbers that Sylvain provided on June 9th  
>>> of this year (see attached),  there was a performance difference  
>>> between mmap on /tmp and mmap on a tmpfs-like file system (/dev/ 
>>> shm in that particular case).  Using the new POSIX component  
>>> provides us with a portable way to provide similar shared memory  
>>> performance gains without having to worry about where the OMPI  
>>> session directory is rooted.
>>>
>>> --
>>> Samuel K. Gutierrez
>>> Los Alamos National Laboratory
>>>
>>> [cid:3364459484_11867134]
>>>
>>>
>>>  I still working on testing the code on Solaris but I don't  
>>> imagine I will see anything that will change my mind.
>>>
>>>  --td
>>>
>>>  Samuel K. Gutierrez wrote:
>>> Hi Rich,
>>>
>>>  It's a modification to the existing common sm component.  The  
>>> modifications do include the addition of a new POSIX shared memory  
>>> facility, however.
>>>
>>>  Sam
>>>
>>>  On Aug 11, 2010, at 10:05 AM, Graham, Richard L. wrote:
>>>
>>>
>>> Is this a modification of the existing component, or a new  
>>> component ?
>>>
>>>  Rich
>>>
>>>
>>>  On 8/10/10 10:52 AM, "Samuel K. Gutierrez" <samuel_at_[hidden]> <mailto:samuel_at_[hidden] 
>>> >  wrote:
>>>
>>>  Hi,
>>>
>>>  I wanted to give everyone a heads-up about a new POSIX shared  
>>> memory
>>>  component
>>>  that has been in the works for a while now and is ready to be  
>>> pushed
>>>  into the
>>>  trunk.
>>>
>>>  http://bitbucket.org/samuelkgutierrez/ompi_posix_sm_new
>>>
>>>  Some highlights:
>>>  o New posix component now the new default.
>>>         o May address some of the shared memory performance issues  
>>> users
>>>  encounter
>>>            when OMPI's session directories are inadvertently  
>>> placed on a non-
>>>  local
>>>            filesystem.
>>>  o Silent component failover.
>>>         o In the default case, if the posix component fails  
>>> initialization,
>>>            mmap will be selected.
>>>  o The sysv component will only be queried for selection if it is
>>>  placed before
>>>     the mmap component (for example, -mca mpi_common_sm
>>>  sysv,posix,mmap).  In the
>>>     default case, sysv will never be queried/selected.
>>>  o Per some on-list discussion, now unlinking mmaped file in both  
>>> mmap
>>>  and posix
>>>     components (see: "System V Shared Memory for Open MPI: Request  
>>> for
>>>  Community
>>>     Input and Testing" thread).
>>>  o  Assuming local process homogeneity with respect to all utilized
>>>  shared
>>>      memory facilities. That is, if one local process deems a
>>>  particular shared
>>>      memory facility acceptable, then ALL local processes should be
>>>  able to
>>>      utilize that facility. As it stands, this is an important point
>>>  because one
>>>      process dictates to all other local processes which common sm
>>>  component will
>>>      be selected based on its own, local run-time test.
>>>  o Addressed some of George's code reuse concerns.
>>>
>>>  If there are no major objections by August 17th, I'll commit the  
>>> code
>>>  after the
>>>  Tuesday morning conference call.
>>>
>>>  Thanks!
>>>
>>>  --
>>>  Samuel K. Gutierrez
>>>  Los Alamos National Laboratory
>>>
>>>
>>>
>>>
>>>
>>>  _______________________________________________
>>>  devel mailing list
>>>  devel_at_[hidden]
>>>  http://www.open-mpi.org/mailman/listinfo.cgi/devel
>>>
>>>
>>>  _______________________________________________
>>>  devel mailing list
>>>  devel_at_[hidden]
>>>  http://www.open-mpi.org/mailman/listinfo.cgi/devel
>>>
>>>
>>>  _______________________________________________
>>>  devel mailing list
>>>  devel_at_[hidden]
>>>  http://www.open-mpi.org/mailman/listinfo.cgi/devel
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> devel mailing list
>>> devel_at_[hidden]
>>> http://www.open-mpi.org/mailman/listinfo.cgi/devel
>>
>>
>> -- 
>> <mime-attachment.gif>
>> Terry D. Dontje | Principal Software Engineer
>> Developer Tools Engineering | +1.650.633.7054
>> Oracle - Performance Technologies
>> 95 Network Drive, Burlington, MA 01803
>> Email terry.dontje_at_[hidden]
>>
>
>
> -- 
> <mime-attachment.gif>
> Terry D. Dontje | Principal Software Engineer
> Developer Tools Engineering | +1.650.633.7054
> Oracle - Performance Technologies
> 95 Network Drive, Burlington, MA 01803
> Email terry.dontje_at_[hidden]
>
> _______________________________________________
> devel mailing list
> devel_at_[hidden]
> http://www.open-mpi.org/mailman/listinfo.cgi/devel