Hello Yevgeny, hello all,
Yevgeny, first of all thanks for explaining what the MTT parameters do and why
there are two of them! I mean this post:
Well, the official recommendation is "twice the RAM amount".
And here we are: we have 2 nodes with 2 TB (that with a 'tera') RAM and a couple
of nodes with 1TB, each with 4x Mellanox IB adapters. Thus we should have raised
the MTT parameters in order to make up to 4 TB memory registrable.
I've tried to raise the MTT parameters in multiple combinations, but the maximum
amount of registrable memory I was able to get was one TB (23 / 5). All tries
to get more (24/5, 23/6 for 2 TB) lead to not responding InfiniBand HCAs.
Is there any another limits in the kernel have to be adjusted in order to be
able to register that a bunch of memory?
Dipl.-Inform. Paul Kapinos - High Performance Computing,
RWTH Aachen University, Center for Computing and Communication
Seffenter Weg 23, D 52074 Aachen (Germany)
Tel: +49 241/80-24915