Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

From: Brock Palen (brockp_at_[hidden])
Date: 2006-12-12 16:29:15


On Dec 12, 2006, at 4:24 PM, Gleb Natapov wrote:

> On Tue, Dec 12, 2006 at 12:58:00PM -0800, Reese Faucette wrote:
>>> Well I have no luck in finding a way to up the amount the system
>>> will
>>> allow GM to use. What is a recommended solution? Is this even a
>>> problem in most cases? Like am i encountering a corner case?
>>
>> upping the limit was not what i'm suggesting as a fix, just
>> pointing out
>> that it is kind of low and even with a fully working ompi or mpich-
>> gm. ompi
>> should still work, even if the IOMMU limit is low.
>>
>> Since you are running 1 thread per CPU (== 2 total), it is
>> possible (likely)
>> that the 1st thread is grabbing all the available registerable
>> memory,
>> leaving not even enough for the second thread to even start. I
>> recommend
>> you try the "mpool_rdma_rcache_size_limit" that Gleb mentions - the
>> equivalent setting is used in MPICH-GM in similar situations. Set
>> this to
>> about 180 MB and run with that.
>>
>> Gleb - I assume that when registration needs exceed
>> "mpool_rdma_rcache_size_limit", that previously registered memory is
>> unregistered much as virtual memory is swapped out?
>>
> If previously registered memory is in use than registration returns
> error to upper layer and operation is retried late. Otherwise
> unused memory
> is unregistered. The code for mpool_rdma_rcache_size_limit is not on
> trunk yet. It is on tmp branch /tmp/gleb-mpool, I don't know if /
> tmp is
> open to everyone. If not I can send the patch.

If you could mail me the patch ill try it out.
>
> --
> Gleb.
> _______________________________________________
> users mailing list
> users_at_[hidden]
> http://www.open-mpi.org/mailman/listinfo.cgi/users
>
>