Open MPI logo

Open MPI Development Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Development mailing list

Subject: Re: [OMPI devel] Trunk Commit Heads-up: New Common Shared Memory Component
From: Terry Dontje (terry.dontje_at_[hidden])
Date: 2010-08-12 07:21:59


Will do.

--td
Samuel K. Gutierrez wrote:
> Hi Terry,
>
> One more thing... Before testing on Solaris 10, could you please
> update (I just committed a Solaris 10 fix).
>
> Thanks,
>
> --
> Samuel K. Gutierrez
> Los Alamos National Laboratory
>
> On Aug 11, 2010, at 1:15 PM, Samuel K. Gutierrez wrote:
>
>> Hi Terry,
>>
>>
>>
>>
>>
>>
>>
>>
>> On Aug 11, 2010, at 12:34 PM, Terry Dontje wrote:
>>
>>> I've done some minor testing on Linux looking at resident and shared
>>> memory sizes for np=4, 8 and 16 jobs. I could not see any
>>> appreciable differences in sizes in the process between sysv, posix
>>> or mmap usage in the SM btl.
>>>
>>> So I am still somewhat non-plussed about making this the default.
>>> It seems like the biggest gain out of using posix might be one
>>> doesn't need to worry about the location of the backing file. This
>>> seems kind of frivolous to me since there is a warning that happens
>>> if the backing file is not memory based.
>>
>> If I'm not mistaken, the warning is only issued if the backing files
>> is stored on the following file systems: Lustre, NFS, Panasas, and
>> GPFS (see: opal_path_nfs in opal/util/path.c). Based on the
>> performance numbers that Sylvain provided on June 9th of this year
>> (see attached), there was a performance difference between mmap on
>> /tmp and mmap on a tmpfs-like file system (/dev/shm in that
>> particular case). Using the new POSIX component provides us with a
>> portable way to provide similar shared memory performance gains
>> without having to worry about where the OMPI session directory is rooted.
>>
>> --
>> Samuel K. Gutierrez
>> Los Alamos National Laboratory
>>
>> <sm-compared.png>
>>
>>>
>>> I still working on testing the code on Solaris but I don't imagine I
>>> will see anything that will change my mind.
>>>
>>> --td
>>>
>>> Samuel K. Gutierrez wrote:
>>>> Hi Rich,
>>>>
>>>> It's a modification to the existing common sm component. The
>>>> modifications do include the addition of a new POSIX shared memory
>>>> facility, however.
>>>>
>>>> Sam
>>>>
>>>> On Aug 11, 2010, at 10:05 AM, Graham, Richard L. wrote:
>>>>
>>>>> Is this a modification of the existing component, or a new
>>>>> component ?
>>>>>
>>>>> Rich
>>>>>
>>>>>
>>>>> On 8/10/10 10:52 AM, "Samuel K. Gutierrez" <samuel_at_[hidden]> wrote:
>>>>>
>>>>> Hi,
>>>>>
>>>>> I wanted to give everyone a heads-up about a new POSIX shared memory
>>>>> component
>>>>> that has been in the works for a while now and is ready to be pushed
>>>>> into the
>>>>> trunk.
>>>>>
>>>>> http://bitbucket.org/samuelkgutierrez/ompi_posix_sm_new
>>>>>
>>>>> Some highlights:
>>>>> o New posix component now the new default.
>>>>> o May address some of the shared memory performance issues
>>>>> users
>>>>> encounter
>>>>> when OMPI's session directories are inadvertently placed
>>>>> on a non-
>>>>> local
>>>>> filesystem.
>>>>> o Silent component failover.
>>>>> o In the default case, if the posix component fails
>>>>> initialization,
>>>>> mmap will be selected.
>>>>> o The sysv component will only be queried for selection if it is
>>>>> placed before
>>>>> the mmap component (for example, -mca mpi_common_sm
>>>>> sysv,posix,mmap). In the
>>>>> default case, sysv will never be queried/selected.
>>>>> o Per some on-list discussion, now unlinking mmaped file in both mmap
>>>>> and posix
>>>>> components (see: "System V Shared Memory for Open MPI: Request for
>>>>> Community
>>>>> Input and Testing" thread).
>>>>> o Assuming local process homogeneity with respect to all utilized
>>>>> shared
>>>>> memory facilities. That is, if one local process deems a
>>>>> particular shared
>>>>> memory facility acceptable, then ALL local processes should be
>>>>> able to
>>>>> utilize that facility. As it stands, this is an important point
>>>>> because one
>>>>> process dictates to all other local processes which common sm
>>>>> component will
>>>>> be selected based on its own, local run-time test.
>>>>> o Addressed some of George's code reuse concerns.
>>>>>
>>>>> If there are no major objections by August 17th, I'll commit the code
>>>>> after the
>>>>> Tuesday morning conference call.
>>>>>
>>>>> Thanks!
>>>>>
>>>>> --
>>>>> Samuel K. Gutierrez
>>>>> Los Alamos National Laboratory
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> devel mailing list
>>>>> devel_at_[hidden]
>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/devel
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> devel mailing list
>>>>> devel_at_[hidden]
>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/devel
>>>>
>>>> _______________________________________________
>>>> devel mailing list
>>>> devel_at_[hidden]
>>>> http://www.open-mpi.org/mailman/listinfo.cgi/devel
>>>
>>>
>>> --
>>> <mime-attachment.gif>
>>> Terry D. Dontje | Principal Software Engineer
>>> Developer Tools Engineering | +1.650.633.7054
>>> Oracle * - Performance Technologies*
>>> 95 Network Drive, Burlington, MA 01803
>>> Email terry.dontje_at_[hidden] <mailto:terry.dontje_at_[hidden]>
>>>
>>> _______________________________________________
>>> devel mailing list
>>> devel_at_[hidden] <mailto:devel_at_[hidden]>
>>> http://www.open-mpi.org/mailman/listinfo.cgi/devel
>>
>> _______________________________________________
>> devel mailing list
>> devel_at_[hidden] <mailto:devel_at_[hidden]>
>> http://www.open-mpi.org/mailman/listinfo.cgi/devel
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> devel mailing list
> devel_at_[hidden]
> http://www.open-mpi.org/mailman/listinfo.cgi/devel

-- 
Oracle
Terry D. Dontje | Principal Software Engineer
Developer Tools Engineering | +1.650.633.7054
Oracle * - Performance Technologies*
95 Network Drive, Burlington, MA 01803
Email terry.dontje_at_[hidden] <mailto:terry.dontje_at_[hidden]>



picture