Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

From: Tim Prins (tprins_at_[hidden])
Date: 2007-08-23 13:40:25


Hi Josh,

I am not an expert in this area of the code, but I'll give it a shot.

(I assume you are using linux due to your email address) When using the memory
manager (which is the default on linux), we wrap malloc/realloc/etc with
ptmalloc2 (which is the same allocator used in glibc 2.3.x).

What I believe is happening is that ptmalloc2 is requesting more memory than
necessary from the OS, and then lazily releasing it back. Try looking in the
ompi source at opal/mca/memory/ptmalloc2/README (
https://svn.open-mpi.org/trac/ompi/browser/tags/v1.2-series/v1.2.3/opal/mca/memory/ptmalloc2/README#L121 ).

This mentions some environment variables that can be set to alter ptmalloc2's
behavior, although I have no idea if they work.

Alternatively, if you are not using a high performance network, there is
little reason to use the memory manager, so you could just disable it.

Tim

On Thursday 23 August 2007 10:18:45 am Josh Aune wrote:
> I have found that the infiniserv MPI that comes with our IB software
> distribution tracks the same behaviour as gcc (releaseing memory on
> realloc). I have also found that building openmpi with
> --without-memory-manager makes openmpi track the same behaviour as
> glibc. I'm guessing that there is a bug in the pinned pages caching
> code?
>
> On 8/21/07, Josh Aune <luken_at_[hidden]> wrote:
> > The realloc included with openmpi 1.2.3 is not releasing memory to the
> > OS and is causing apps to go into swap. Attached is a little test
> > program that shows calls to realloc not releasing the memory when
> > compiled using mpicc, but when compiled directly with gcc (or icc)
> > calling realloc() frees any memory no longer needed.
> >
> > Is this a bug?
> >
> > If not, how can I force openmpi to free the memory that the allocator
> > is sitting on?
> >
> > Thanks,
> > Josh
> >
> > Sample output. Note the delta between 'total' and 'malloc held' when
> > compiled with mpicc and how the gcc compiled versions track perfectly.
> >
> > $ mpicc -o realloc_test realloc_test.c
> > $ ./realloc_test
> > ...
> > malloc/realloc/free test
> > malloc() 50 MB, total 50 MB, malloc held 50 MB
> > realloc() 1 MB, total 1 MB, malloc held 50 MB
> > malloc() 50 MB, total 51 MB, malloc held 100 MB
> > realloc() 1 MB, total 2 MB, malloc held 100 MB
> > malloc() 50 MB, total 52 MB, malloc held 150 MB
> > realloc() 1 MB, total 3 MB, malloc held 150 MB
> > malloc() 50 MB, total 53 MB, malloc held 200 MB
> > realloc() 1 MB, total 4 MB, malloc held 200 MB
> > malloc() 50 MB, total 54 MB, malloc held 250 MB
> > realloc() 1 MB, total 5 MB, malloc held 250 MB
> > free() 1 MB, total 4 MB, malloc held 200 MB
> > free() 1 MB, total 3 MB, malloc held 150 MB
> > free() 1 MB, total 2 MB, malloc held 100 MB
> > free() 1 MB, total 1 MB, malloc held 50 MB
> > free() 1 MB, total 0 MB, malloc held 0 MB
> > ...
> >
> > $ gcc -o realloc_test realloc_test.c
> > $ ./realloc_test
> > ...
> > malloc/realloc/free test
> > malloc() 50 MB, total 50 MB, malloc held 50 MB
> > realloc() 1 MB, total 1 MB, malloc held 1 MB
> > malloc() 50 MB, total 51 MB, malloc held 51 MB
> > realloc() 1 MB, total 2 MB, malloc held 2 MB
> > malloc() 50 MB, total 52 MB, malloc held 52 MB
> > realloc() 1 MB, total 3 MB, malloc held 3 MB
> > malloc() 50 MB, total 53 MB, malloc held 53 MB
> > realloc() 1 MB, total 4 MB, malloc held 4 MB
> > malloc() 50 MB, total 54 MB, malloc held 54 MB
> > realloc() 1 MB, total 5 MB, malloc held 5 MB
> > free() 1 MB, total 4 MB, malloc held 4 MB
> > free() 1 MB, total 3 MB, malloc held 3 MB
> > free() 1 MB, total 2 MB, malloc held 2 MB
> > free() 1 MB, total 1 MB, malloc held 1 MB
> > free() 1 MB, total 0 MB, malloc held 0 MB
> > ...
>
> _______________________________________________
> users mailing list
> users_at_[hidden]
> http://www.open-mpi.org/mailman/listinfo.cgi/users