I would start by adjusting btl_openib_receive_queues . The default uses a per-peer QP which can eat up a lot of memory. I recommend using no per-peer and several shared receive queues. We use S,4096,1024:S,12288,512:S,65536,512
On Thu, 12 Jan 2012, V. Ram wrote:
> Open MPI IB Gurus,
> I have some slightly older InfiniBand-equipped nodes with IB which have
> less RAM than we'd like, and on which we tend to run jobs that can span
> 16-32 nodes of this type. The jobs themselves tend to run on the heavy
> side in terms of their own memory requirements.
> When we used to run on an older Intel MPI, these jobs managed to run
> within the available RAM without paging out to disk. Now using Open MPI
> 1.5.3, we can end up paging to disk or even running out of memory for
> the same codes and exact same jobs and node distributions.
> I'm suspecting that I can reduce overall memory consumption by tuning
> the IB-related memory that Open MPI consumes. I've looked at the FAQ:
> , but I'm still not certain about where I should start. Again, this is
> all for 1.5.3 (we are willing to update to 1.5.4 or 1.5.5 when released,
> if it would help).
> 1. It looks like there are several independent IB BTL MCA parameters to
> try adjusting: i. mpool_rdma_rcache_size_limit, ii.
> btl_openib_free_list_max , iii. btl_openib_max_send_size , iv.
> btl_openib_eager_rdma_num, v. btl_openib_max_eager_rdma, vi.
> btl_openib_eager_limit . Have I missed any others parameters that
> impact InfiniBand-related memory usage? These parameters are listed as
> affecting registered memory. Are there parameters that affect
> unregistered IB-related memory consumption on the part of Open MPI
> 2. Where should I start with this? For example, is it worth trying to
> adjust any of the eager parameters, or are the bulk of the memory
> requirements coming from the mpool_rdma_rcache_size_limit?
> 3. Are there any gross/overall "master" parameters that will set limits,
> but keep the various buffers in intelligent proportion to one another,
> or will I need to manually adjust each set of buffers independently? If
> the latter, are there any guidelines on the relative proportions between
> buffers, or overall recommendations?
> Thank you very much.
> http://www.fastmail.fm - A fast, anti-spam email service.
> users mailing list