Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: Re: [OMPI users] Shared Memory Performance Problem.
From: Tim Prince (n8tm_at_[hidden])
Date: 2011-03-27 09:37:53

On 3/27/2011 2:26 AM, Michele Marena wrote:
> Hi,
> My application performs good without shared memory utilization, but with
> shared memory I get performance worst than without of it.
> Do I make a mistake? Don't I pay attention to something?
> I know OpenMPI uses /tmp directory to allocate shared memory and it is
> in the local filesystem.

I guess you mean shared memory message passing. Among relevant
parameters may be the message size where your implementation switches
from cached copy to non-temporal (if you are on a platform where that
terminology is used). If built with Intel compilers, for example, the
copy may be performed by intel_fast_memcpy, with a default setting which
uses non-temporal when the message exceeds about some preset size, e.g.
50% of smallest L2 cache for that architecture.
A quick search for past posts seems to indicate that OpenMPI doesn't
itself invoke non-temporal, but there appear to be several useful
articles not connected with OpenMPI.
In case guesses aren't sufficient, it's often necessary to profile
(gprof, oprofile, Vtune, ....) to pin this down.
If shared message slows your application down, the question is whether
this is due to excessive eviction of data from cache; not a simple
question, as most recent CPUs have 3 levels of cache, and your
application may require more or less data which was in use prior to the
message receipt, and may use immediately only a small piece of a large

Tim Prince