Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: Re: [OMPI users] Valgrind Functionality
From: Jeff Squyres (jsquyres_at_[hidden])
Date: 2008-07-14 07:40:10

On Jul 14, 2008, at 7:07 AM, Ashley Pittman wrote:

>> The default in v1.3 is that ptmalloc2 is *not* built into libopen-
>> pal. This is different than v1.2, where ptmalloc2 *was* included in
>> libopen-pal unless you specified --disable-memory-manager.
> Thank you for clearing that ip Jeff, what is the cost of using this
> option. The comments in the code led me to believe this was more to
> do
> with pinning memory than anything else?
> Would it be advisable to add a mpicc option to enable and disable
> linking this library, with 1.2.6 I was sucesfully able to compile and
> run a application without it by simply changing the gcc compile line.

FWIW, we have an open ticket to document this stuff before the v1.3
release. :-)

Here's the scoop:

- setting "mpi_leave_pinned" to true does a few things; see
  for details

- we decided that ptmalloc2 is more trouble than it's worth; it causes
real problems in some applications

- but some applications definitely do benefit from leave_pinned behavior

- some kind of memory manager is *required* in OMPI for leave_pinned
behavior to work (i.e., OMPI *has* to have visibility to when memory
is returned to the OS so that we know to invalidate relevant entries
in our internal registration cache)

- there's another option, however: simply just don't return memory to
the OS (via a mallopt hint). This avoids the problem altogether.

- it is important to note that OMPI's ptmalloc2 was configured
essentially to do this anyway (rarely to return memory to the OS), and
we haven't seen complaints about this effect.

- so for v1.3, we have changed the default behavior:
     - when transports that benefits from leave_pinned behavior are
used in a job, mpi_leave_pinned will automatically enable itself
     - OMPI's internal ptmalloc2 library will now be a separate
library (-lopenmpi-malloc) and it will *NOT* be linked in by default.
If you want to use it, you need to add -lopenmpi-malloc to your
wrapper compiler command line.
     - if ptmalloc2 is not available and a transport that would
benefit from leave_pinned is used, we'll use the mallopt hint to
disable memory from being returned to the OS
     - if mallopt fails for some reason, we'll disable leave_pinned

Note that the "automatically enable itself" part isn't enabled yet;
that's still on my to-do list. It'll be done by v1.3. ptmalloc2 as a
separate library is already done.

What this means is quite simple:

- if you run with openfabrics networks, mpi_leave_pinned will now be
the default, but through the mallopt mechanism, not ptmalloc2
- if you run with openfabrics and want to use ptmalloc2, add -lopenmpi-
malloc to your link command line
- if you do not run with openfabrics networks, you're not really
affected (aside from now returning memory to the OS via the normal
libc memory allocator)

Jeff Squyres
Cisco Systems