Open MPI logo

Open MPI Development Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Development mailing list

Subject: Re: [OMPI devel] kernel 2.6.23 vs 2.6.24 - communication/wait times
From: Eugene Loh (eugene.loh_at_[hidden])
Date: 2010-04-19 12:59:38

Ralph Castain wrote:
On Apr 19, 2010, at 9:12 AM, Oliver Geisler wrote:
Ah, that could do it.  Open MPI's shared memory files are under /tmp.  So if /tmp is NFS, you could get extremely high latencies because of dirty page writes out through NFS.

You don't necessarily have to make /tmp disk-full -- if you just make OMPI's session directories go into a ramdisk instead of to NFS, that should also be sufficient.
I just browsed FAQ and "ompi_info --param all all", but didn't find the
How do I set the OMPI session directory to point it to a ramdisk?
Set the MCA param orte_tmpdir_base to point at the ramdisk using any of the MCA parameter methods (cmd line, envar, default mca param file).
I'll add that to

And another question:
What would be a good size for the ram disk? One general value was
supposed by the FAQ with 128MB, but what is your experience?
(maybe a large topic by itself, so I have to try out, I guess)

I don't recall the default shared memory size per process, but you can get that value from ompi_info --param btl sm. Take that value, multiply by your expected ppn, and then give yourself a fudge factor.
Sizing proportionately to the number of processes was a poor heuristic and starting in 1.3.2 we don't employ it any more.  In all likelihood, the default size of the shared-memory backing file will be set by mpool_sm_min_size... 64 Mbytes.  Try "ompi_info --param mpool sm".  There's some other stuff in addition to this backing file...  so, need a little fudge factor.  Probably 128 MB will be enough for the shared-memory stuff.