Open MPI IB Gurus,
I have some slightly older InfiniBand-equipped nodes with IB which have
less RAM than we'd like, and on which we tend to run jobs that can span
16-32 nodes of this type. The jobs themselves tend to run on the heavy
side in terms of their own memory requirements.
When we used to run on an older Intel MPI, these jobs managed to run
within the available RAM without paging out to disk. Now using Open MPI
1.5.3, we can end up paging to disk or even running out of memory for
the same codes and exact same jobs and node distributions.
I'm suspecting that I can reduce overall memory consumption by tuning
the IB-related memory that Open MPI consumes. I've looked at the FAQ:
, but I'm still not certain about where I should start. Again, this is
all for 1.5.3 (we are willing to update to 1.5.4 or 1.5.5 when released,
if it would help).
1. It looks like there are several independent IB BTL MCA parameters to
try adjusting: i. mpool_rdma_rcache_size_limit, ii.
btl_openib_free_list_max , iii. btl_openib_max_send_size , iv.
btl_openib_eager_rdma_num, v. btl_openib_max_eager_rdma, vi.
btl_openib_eager_limit . Have I missed any others parameters that
impact InfiniBand-related memory usage? These parameters are listed as
affecting registered memory. Are there parameters that affect
unregistered IB-related memory consumption on the part of Open MPI
2. Where should I start with this? For example, is it worth trying to
adjust any of the eager parameters, or are the bulk of the memory
requirements coming from the mpool_rdma_rcache_size_limit?
3. Are there any gross/overall "master" parameters that will set limits,
but keep the various buffers in intelligent proportion to one another,
or will I need to manually adjust each set of buffers independently? If
the latter, are there any guidelines on the relative proportions between
buffers, or overall recommendations?
Thank you very much.
http://www.fastmail.fm - A fast, anti-spam email service.