To tell you all what noone wanted to tell me, yes, it does seem to be
the memory manager. Compiling everything with
--with-memory-manager=none returns the vmem use to the more reasonable
~100MB per process (down from >8GB).
I take it this may affect my peak bandwidth over infiniband. What's the
general feeling about how bad this is?
On Tue, 2008-05-13 at 13:12 +1000, Terry Frankcombe wrote:
> Hi folks
> I'm trying to run an MPI app on an infiniband cluster with OpenMPI
> When run on a single node, this app is grabbing large chunks of memory
> (total per process ~8.5GB, including strace showing a single 4GB grab)
> but not using it. The resident memory use is ~40MB per process. When
> this app is compiled in serial mode (with conditionals to remove the MPI
> calls) the memory use is more like what you'd expect, 40MB res and
> ~100MB vmem.
> Now I didn't write it so I'm not sure what extra stuff the MPI version
> does, and we haven't tracked down the large memory grabs.
> Could it be that this vmem is being grabbed by the OpenMPI memory
> manager rather than directly by the app?