Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

From: Reese Faucette (reese_at_[hidden])
Date: 2006-12-12 15:58:00

> Well I have no luck in finding a way to up the amount the system will
> allow GM to use. What is a recommended solution? Is this even a
> problem in most cases? Like am i encountering a corner case?

upping the limit was not what i'm suggesting as a fix, just pointing out
that it is kind of low and even with a fully working ompi or mpich-gm. ompi
should still work, even if the IOMMU limit is low.

Since you are running 1 thread per CPU (== 2 total), it is possible (likely)
that the 1st thread is grabbing all the available registerable memory,
leaving not even enough for the second thread to even start. I recommend
you try the "mpool_rdma_rcache_size_limit" that Gleb mentions - the
equivalent setting is used in MPICH-GM in similar situations. Set this to
about 180 MB and run with that.

Gleb - I assume that when registration needs exceed
"mpool_rdma_rcache_size_limit", that previously registered memory is
unregistered much as virtual memory is swapped out?