I have opened a ticket at https://svn.open-mpi.org/trac/ompi/ticket/1449
which proposes increasing the maximum default size for the sm btl file.
Currently, the file size is 32Mbytes per process with a maximum of
512Mbytes. This means that after np=16, the backing file no longer grows.
We have seen that on some larger SMPs with the potential to run np=512
that this maximum size is too small. We can run out of memory and the
program can fail, or more recently, hang. I also think that 512Mbytes
is just too small a number when you are talking about an SMP with many
gigabytes of /tmp space.
I propose bumping the max for 32-bit programs to 2G and for 64-bit
programs to 8G.
If any one has objections to this, please comment on the ticket where
there are some additional details.