Open MPI logo

Open MPI Development Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Development mailing list

Subject: Re: [OMPI devel] SM backing file size
From: Eugene Loh (Eugene.Loh_at_[hidden])
Date: 2008-11-14 20:31:04


Ralph Castain wrote:

> I have two examples so far:
>
> 1. using a ramdisk, /tmp was set to 10MB. OMPI was run on a single
> node, 2ppn, with btl=openib,sm,self. The program started, but
> segfaulted on the first MPI_Send. No warnings were printed.

Interesting. So far as I can tell, the actual memory consumption (total
number of allocations in the mmapped segment) for 2 local processes
should be a little more than half a Mbyte. The bulk of that would be
from fragments (chunks). There are btl_sm_free_list_num=8 per process,
each of btl_sm_max_frag_size=32K. So, that's 8x2x32K=512Kbyte.
Actually, a little bit more. Anyhow, that accounts for most of the
allocations, I think. Maybe if you're sending a lot of data, more gets
allocated at MPI_Send time. I don't know.

While only < 1 Mbyte is needed, however, mpool_sm_min_size=128M will be
mapped.

It doesn't make sense that this case would fail, but the next case
should run. Are you sure this is related to the SM backing file?

> 2. again with a ramdisk, /tmp was reportedly set to 16MB (unverified
> - some uncertainty, could be have been much larger). OMPI was run on
> multiple nodes, 16ppn, with btl=openib,sm,self. The program ran to
> completion without errors or warning. I don't know the communication
> pattern - could be no local comm was performed, though that sounds
> doubtful.