On Jul 14, 2008, at 7:07 AM, Ashley Pittman wrote:
>> The default in v1.3 is that ptmalloc2 is *not* built into libopen-
>> pal. This is different than v1.2, where ptmalloc2 *was* included in
>> libopen-pal unless you specified --disable-memory-manager.
>
> Thank you for clearing that ip Jeff, what is the cost of using this
> option. The comments in the code led me to believe this was more to
> do
> with pinning memory than anything else?
>
> Would it be advisable to add a mpicc option to enable and disable
> linking this library, with 1.2.6 I was sucesfully able to compile and
> run a application without it by simply changing the gcc compile line.
FWIW, we have an open ticket to document this stuff before the v1.3
release. :-)
Here's the scoop:
- setting "mpi_leave_pinned" to true does a few things; see http://www.open-mpi.org/faq/?category=openfabrics#large-message-leave-pinned
for details
- we decided that ptmalloc2 is more trouble than it's worth; it causes
real problems in some applications
- but some applications definitely do benefit from leave_pinned behavior
- some kind of memory manager is *required* in OMPI for leave_pinned
behavior to work (i.e., OMPI *has* to have visibility to when memory
is returned to the OS so that we know to invalidate relevant entries
in our internal registration cache)
- there's another option, however: simply just don't return memory to
the OS (via a mallopt hint). This avoids the problem altogether.
- it is important to note that OMPI's ptmalloc2 was configured
essentially to do this anyway (rarely to return memory to the OS), and
we haven't seen complaints about this effect.
- so for v1.3, we have changed the default behavior:
- when transports that benefits from leave_pinned behavior are
used in a job, mpi_leave_pinned will automatically enable itself
- OMPI's internal ptmalloc2 library will now be a separate
library (-lopenmpi-malloc) and it will *NOT* be linked in by default.
If you want to use it, you need to add -lopenmpi-malloc to your
wrapper compiler command line.
- if ptmalloc2 is not available and a transport that would
benefit from leave_pinned is used, we'll use the mallopt hint to
disable memory from being returned to the OS
- if mallopt fails for some reason, we'll disable leave_pinned
Note that the "automatically enable itself" part isn't enabled yet;
that's still on my to-do list. It'll be done by v1.3. ptmalloc2 as a
separate library is already done.
What this means is quite simple:
- if you run with openfabrics networks, mpi_leave_pinned will now be
the default, but through the mallopt mechanism, not ptmalloc2
- if you run with openfabrics and want to use ptmalloc2, add -lopenmpi-
malloc to your link command line
- if you do not run with openfabrics networks, you're not really
affected (aside from now returning memory to the OS via the normal
libc memory allocator)
--
Jeff Squyres
Cisco Systems
|
![[Flag]](../../../../../community/mirrors/flag-us.gif){kind=small}