Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |  

This web mail archive is frozen.

This page is part of a frozen web archive of this mailing list.

You can still navigate around this archive, but know that no new mails have been added to it since July of 2016.

Click here to be taken to the new web archives of this list; it includes all the mails that are in this frozen archive plus all new mails that have been sent to the list since it was migrated to the new archives.

From: Gleb Natapov (glebn_at_[hidden])
Date: 2006-12-12 16:24:27

On Tue, Dec 12, 2006 at 12:58:00PM -0800, Reese Faucette wrote:
> > Well I have no luck in finding a way to up the amount the system will
> > allow GM to use. What is a recommended solution? Is this even a
> > problem in most cases? Like am i encountering a corner case?
> upping the limit was not what i'm suggesting as a fix, just pointing out
> that it is kind of low and even with a fully working ompi or mpich-gm. ompi
> should still work, even if the IOMMU limit is low.
> Since you are running 1 thread per CPU (== 2 total), it is possible (likely)
> that the 1st thread is grabbing all the available registerable memory,
> leaving not even enough for the second thread to even start. I recommend
> you try the "mpool_rdma_rcache_size_limit" that Gleb mentions - the
> equivalent setting is used in MPICH-GM in similar situations. Set this to
> about 180 MB and run with that.
> Gleb - I assume that when registration needs exceed
> "mpool_rdma_rcache_size_limit", that previously registered memory is
> unregistered much as virtual memory is swapped out?
If previously registered memory is in use than registration returns
error to upper layer and operation is retried late. Otherwise unused memory
is unregistered. The code for mpool_rdma_rcache_size_limit is not on
trunk yet. It is on tmp branch /tmp/gleb-mpool, I don't know if /tmp is
open to everyone. If not I can send the patch.