On 13-Jan-12 12:23 AM, Nathan Hjelm wrote:
> I would start by adjusting btl_openib_receive_queues . The default uses
> a per-peer QP which can eat up a lot of memory. I recommend using no
> per-peer and several shared receive queues.
> We use S,4096,1024:S,12288,512:S,65536,512
And here's the FAQ entry that explains the logic behind this voodoo option:
> On Thu, 12 Jan 2012, V. Ram wrote:
>> Open MPI IB Gurus,
>> I have some slightly older InfiniBand-equipped nodes with IB which have
>> less RAM than we'd like, and on which we tend to run jobs that can span
>> 16-32 nodes of this type. The jobs themselves tend to run on the heavy
>> side in terms of their own memory requirements.
>> When we used to run on an older Intel MPI, these jobs managed to run
>> within the available RAM without paging out to disk. Now using Open MPI
>> 1.5.3, we can end up paging to disk or even running out of memory for
>> the same codes and exact same jobs and node distributions.
>> I'm suspecting that I can reduce overall memory consumption by tuning
>> the IB-related memory that Open MPI consumes. I've looked at the FAQ:
>> , but I'm still not certain about where I should start. Again, this is
>> all for 1.5.3 (we are willing to update to 1.5.4 or 1.5.5 when released,
>> if it would help).
>> 1. It looks like there are several independent IB BTL MCA parameters to
>> try adjusting: i. mpool_rdma_rcache_size_limit, ii.
>> btl_openib_free_list_max , iii. btl_openib_max_send_size , iv.
>> btl_openib_eager_rdma_num, v. btl_openib_max_eager_rdma, vi.
>> btl_openib_eager_limit . Have I missed any others parameters that
>> impact InfiniBand-related memory usage? These parameters are listed as
>> affecting registered memory. Are there parameters that affect
>> unregistered IB-related memory consumption on the part of Open MPI
>> 2. Where should I start with this? For example, is it worth trying to
>> adjust any of the eager parameters, or are the bulk of the memory
>> requirements coming from the mpool_rdma_rcache_size_limit?
>> 3. Are there any gross/overall "master" parameters that will set limits,
>> but keep the various buffers in intelligent proportion to one another,
>> or will I need to manually adjust each set of buffers independently? If
>> the latter, are there any guidelines on the relative proportions between
>> buffers, or overall recommendations?
>> Thank you very much.
>> http://www.fastmail.fm - A fast, anti-spam email service.
>> users mailing list
> users mailing list