Open MPI logo

Open MPI Development Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Development mailing list

Subject: Re: [OMPI devel] Trunk Commit Heads-up: New Common Shared Memory Component
From: Terry Dontje (terry.dontje_at_[hidden])
Date: 2010-08-12 07:21:23


Samuel K. Gutierrez wrote:
>
> If I'm not mistaken, the warning is only issued if the backing files
> is stored on the following file systems: Lustre, NFS, Panasas, and
> GPFS (see: opal_path_nfs in opal/util/path.c). Based on the
> performance numbers that Sylvain provided on June 9th of this year
> (see attached), there was a performance difference between mmap on
> /tmp and mmap on a tmpfs-like file system (/dev/shm in that particular
> case). Using the new POSIX component provides us with a portable way
> to provide similar shared memory performance gains without having to
> worry about where the OMPI session directory is rooted.
>
Is there not a way to determine whether the fs is tmpfs or not? It
seems fixing that is a lot less changes then adapting to Posix shared
memory. My main concern is we have had quite a long runtime with the
mmap implementation and know it works for us. We haven't had nearly as
much runtime with this Posix implementation and so who knows what issues
we might run into. Note, I am not saying Posix is broken I am just
saying we have not used it within our btl as much and who knows what we
might run into (or not).

To me the fs homing issue seems like a weak case. I understand it is an
out of the box feature and nicety but I just wonder if that weakness is
better solved without replacing the shared memory calls we use. Note, I
am on the fence to some extent on this. I understand that we could give
this feature a lot of soak time and probably come to a reasonable belief
we haven't gone backwards somehow but it just seems like there is very
little gain in this IMO.

Am I in the minority on the above belief?

--td

> --
> Samuel K. Gutierrez
> Los Alamos National Laboratory
>
>
>>
>> I still working on testing the code on Solaris but I don't imagine I
>> will see anything that will change my mind.
>>
>> --td
>>
>> Samuel K. Gutierrez wrote:
>>> Hi Rich,
>>>
>>> It's a modification to the existing common sm component. The
>>> modifications do include the addition of a new POSIX shared memory
>>> facility, however.
>>>
>>> Sam
>>>
>>> On Aug 11, 2010, at 10:05 AM, Graham, Richard L. wrote:
>>>
>>>> Is this a modification of the existing component, or a new component ?
>>>>
>>>> Rich
>>>>
>>>>
>>>> On 8/10/10 10:52 AM, "Samuel K. Gutierrez" <samuel_at_[hidden]> wrote:
>>>>
>>>> Hi,
>>>>
>>>> I wanted to give everyone a heads-up about a new POSIX shared memory
>>>> component
>>>> that has been in the works for a while now and is ready to be pushed
>>>> into the
>>>> trunk.
>>>>
>>>> http://bitbucket.org/samuelkgutierrez/ompi_posix_sm_new
>>>>
>>>> Some highlights:
>>>> o New posix component now the new default.
>>>> o May address some of the shared memory performance issues
>>>> users
>>>> encounter
>>>> when OMPI's session directories are inadvertently placed
>>>> on a non-
>>>> local
>>>> filesystem.
>>>> o Silent component failover.
>>>> o In the default case, if the posix component fails
>>>> initialization,
>>>> mmap will be selected.
>>>> o The sysv component will only be queried for selection if it is
>>>> placed before
>>>> the mmap component (for example, -mca mpi_common_sm
>>>> sysv,posix,mmap). In the
>>>> default case, sysv will never be queried/selected.
>>>> o Per some on-list discussion, now unlinking mmaped file in both mmap
>>>> and posix
>>>> components (see: "System V Shared Memory for Open MPI: Request for
>>>> Community
>>>> Input and Testing" thread).
>>>> o Assuming local process homogeneity with respect to all utilized
>>>> shared
>>>> memory facilities. That is, if one local process deems a
>>>> particular shared
>>>> memory facility acceptable, then ALL local processes should be
>>>> able to
>>>> utilize that facility. As it stands, this is an important point
>>>> because one
>>>> process dictates to all other local processes which common sm
>>>> component will
>>>> be selected based on its own, local run-time test.
>>>> o Addressed some of George's code reuse concerns.
>>>>
>>>> If there are no major objections by August 17th, I'll commit the code
>>>> after the
>>>> Tuesday morning conference call.
>>>>
>>>> Thanks!
>>>>
>>>> --
>>>> Samuel K. Gutierrez
>>>> Los Alamos National Laboratory
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> devel mailing list
>>>> devel_at_[hidden]
>>>> http://www.open-mpi.org/mailman/listinfo.cgi/devel
>>>>
>>>>
>>>> _______________________________________________
>>>> devel mailing list
>>>> devel_at_[hidden]
>>>> http://www.open-mpi.org/mailman/listinfo.cgi/devel
>>>
>>> _______________________________________________
>>> devel mailing list
>>> devel_at_[hidden]
>>> http://www.open-mpi.org/mailman/listinfo.cgi/devel
>>
>>
>> --
>> <mime-attachment.gif>
>> Terry D. Dontje | Principal Software Engineer
>> Developer Tools Engineering | +1.650.633.7054
>> Oracle * - Performance Technologies*
>> 95 Network Drive, Burlington, MA 01803
>> Email terry.dontje_at_[hidden] <mailto:terry.dontje_at_[hidden]>
>>
>> _______________________________________________
>> devel mailing list
>> devel_at_[hidden] <mailto:devel_at_[hidden]>
>> http://www.open-mpi.org/mailman/listinfo.cgi/devel
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> devel mailing list
> devel_at_[hidden]
> http://www.open-mpi.org/mailman/listinfo.cgi/devel

-- 
Oracle
Terry D. Dontje | Principal Software Engineer
Developer Tools Engineering | +1.650.633.7054
Oracle * - Performance Technologies*
95 Network Drive, Burlington, MA 01803
Email terry.dontje_at_[hidden] <mailto:terry.dontje_at_[hidden]>



picture
picture