On Tue, Dec 12, 2006 at 12:58:00PM -0800, Reese Faucette wrote:
> > Well I have no luck in finding a way to up the amount the system will
> > allow GM to use. What is a recommended solution? Is this even a
> > problem in most cases? Like am i encountering a corner case?
> upping the limit was not what i'm suggesting as a fix, just pointing out
> that it is kind of low and even with a fully working ompi or mpich-gm. ompi
> should still work, even if the IOMMU limit is low.
> Since you are running 1 thread per CPU (== 2 total), it is possible (likely)
> that the 1st thread is grabbing all the available registerable memory,
> leaving not even enough for the second thread to even start. I recommend
> you try the "mpool_rdma_rcache_size_limit" that Gleb mentions - the
> equivalent setting is used in MPICH-GM in similar situations. Set this to
> about 180 MB and run with that.
> Gleb - I assume that when registration needs exceed
> "mpool_rdma_rcache_size_limit", that previously registered memory is
> unregistered much as virtual memory is swapped out?
If previously registered memory is in use than registration returns
error to upper layer and operation is retried late. Otherwise unused memory
is unregistered. The code for mpool_rdma_rcache_size_limit is not on
trunk yet. It is on tmp branch /tmp/gleb-mpool, I don't know if /tmp is
open to everyone. If not I can send the patch.