Open MPI logo

Open MPI Development Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Development mailing list

Subject: Re: [OMPI devel] 1.6.1rc1 posted
From: Paul Kapinos (kapinos_at_[hidden])
Date: 2012-08-14 11:26:45


Hi Jeff,
Hi All,

On 08/07/12 18:51, Jeff Squyres wrote:
> So I'm not 100% clear on what you mean here: when you set the OFED params to
> allow registration of more memory than you have physically,
> does the problem go away?

We are talking about machines with 24GB RAM (S) and 96GB RAM (L).
The default values for Mellanox/OFED parameter are 20/3 => 32GB registerable
memory (RM) on both S and L. This is more than memory of S, but less than 2x
memory of S, and less than memory of L.

If the OFED parameter are pimped to at least RM=64GB (20/3 => 21/3, 22/3, 24/3)
there are no errors, I've just tested it with 8GB respectively 15.5 GB of data
(starting usually 1x ppn).

If the OFED parameter are _not_ changed (=32GB RM) there is _no_ warning on S
nodes; on L nodes this warns the user:
--------------------------------------------------------------------------
WARNING: It appears that your OpenFabrics subsystem is configured to only
.......
   Registerable memory: 32768 MiB
   Total memory: 98293 MiB
--------------------------------------------------------------------------
.. hardly surprising - the warning came if and only if (RM < memory).

If the OFED parameter are _not_ changed (=32GB RM) and I'm trying to send at
least 8GB _in one chunk_ then the 'queue pair' error came out (see S_log.txt and
my last mail). More exactly at least one process seem to die in MPI_Finalize
(all output of the program is correct). The same error came out also on L nodes,
surrounded by the above warning (L_log.txt).

>> From your log messages, the warning messages were from machines with
>> nearly 100GB RAM but only 32GB register-able. But only one of those was
>> the same as one that showed QP creation failures.
>> So I'm not clear which was which.
>
> Regardless: can you pump the MTT params up to allow registering all
>> of physical memory on those machines, and see if you get any failures?

as you can see on a node with 24GB memory and 32GB RM there can be a failure
without any warning from Open MPI side :-(

> To be clear: we're trying to determine if we should spend more effort
> on making OMPI work properly in low-registered-memory-availabile
> scenarios, or whether we should just emphasize
> "go reset your MTT parameters to allow registering all of physical memory."

After making the experience with failures when only 1.5x of phys.mem. is allowed
for registering I would follow Mellanox in "go reset your MTT to allow _twice_
the phys.memory".

So,
- if the OFED parameter are pimped everything is OK
- there is a [rare] combination when your great workaround did not catch.
- allowing 2x memory for being registered could be a good idea.

Does this make sense?

Best,
Paul Kapinos

P.S. The used example program is of course an synthetical thing but it is
strongly sympathized with the Serpent software. (however serpent usually use
chunks whereby the actual error arise if all the 8GB are send in one piece).

P.S.2 When all works, with increasing chunk size to HUGE values, the performance
seem to became worse - sending all 15.5 GB in one piece is more than twice
slower than sending with 200 mb pieces. See chunked_send.txt
(the first parameter is #doubles of data, the 2nd is #doubles in a chunk).

P.S.3 all experiments above with 1.6.1rc2

P.S.4. I'm also performing some linpack runs with 6x nodes and my very first
impression is that increasing log_num_mtt to huge values is a bad idea
(performance loss of some 5%). But let me finish it first...

-- 
Dipl.-Inform. Paul Kapinos   -   High Performance Computing,
RWTH Aachen University, Center for Computing and Communication
Seffenter Weg 23,  D 52074  Aachen (Germany)
Tel: +49 241/80-24915