Which version of Open MPI are you using ? We can figure out what's
wrong if we have the output of "ompi_info" and "ompi_info --param all
I wonder if some of the memory is not related to the size of the
shared memory file. The default way to compute the size of the shared
memory file is defined by the MCA parameter mpool_sm_per_peer_size.
By default is set to 128MB for each local peer. Therefore using 2048
procs on 256 nodes lead to using 8 procs by node i.e. at least 1GB
only for the SM file. The problem right now with the SM file is that
we're not reusing the buffers multiple times, instead we're using a
new fragment each time we send a message, forcing the OS to map the
entire file at one point.
On Nov 27, 2006, at 8:21 PM, Matt Leininger wrote:
> On Mon, 2006-11-27 at 16:45 -0800, Matt Leininger wrote:
>> Has anyone testing OMPI's alltoall at > 2000 MPI tasks? I'm
>> seeing each
>> MPI task eat up > 1GB of memory (just for OMPI - not the app).
> I gathered some more data using the alltoall benchmark in mpiBench.
> mpiBench is pretty smart about how large its buffers are. I set it to
> use <= 100MB.
> num nodes num MPI tasks system mem mpibench buffer mem
> 128 1024 1 GB 65 MB
> 160 1280 1.2 GB 82 MB
> 192 1536 1.4 GB 98 MB
> 224 1792 1.6 GB 57 MB
> 256 2048 1.6-1.8 GB < 100 MB
> The 256 node run was killed by the OOM for using too much memory. For
> all these tests the OMPI alltoall is using 1 GB or more of system
> memory. I know LANL is looking into optimized alltoall, but is anyone
> looking into the scalability of the memory footprint?
> - Matt
>> - Matt
>> devel mailing list
> devel mailing list