Open MPI logo

Open MPI Development Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Development mailing list

Subject: [OMPI devel] MTT parameters vor really big nodes?
From: Paul Kapinos (kapinos_at_[hidden])
Date: 2012-10-31 16:22:27


Hello Yevgeny, hello all,

Yevgeny, first of all thanks for explaining what the MTT parameters do and why
there are two of them! I mean this post:
http://www.open-mpi.org/community/lists/devel/2012/08/11417.php

Well, the official recommendation is "twice the RAM amount".

And here we are: we have 2 nodes with 2 TB (that with a 'tera') RAM and a couple
of nodes with 1TB, each with 4x Mellanox IB adapters. Thus we should have raised
the MTT parameters in order to make up to 4 TB memory registrable.

I've tried to raise the MTT parameters in multiple combinations, but the maximum
amount of registrable memory I was able to get was one TB (23 / 5). All tries
to get more (24/5, 23/6 for 2 TB) lead to not responding InfiniBand HCAs.

Is there any another limits in the kernel have to be adjusted in order to be
able to register that a bunch of memory?

Best,

Paul Kapinos

-- 
Dipl.-Inform. Paul Kapinos   -   High Performance Computing,
RWTH Aachen University, Center for Computing and Communication
Seffenter Weg 23,  D 52074  Aachen (Germany)
Tel: +49 241/80-24915