Open MPI logo

Open MPI Development Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Development mailing list

Subject: Re: [OMPI devel] 1.6.1rc1 posted
From: Jeff Squyres (jsquyres_at_[hidden])
Date: 2012-08-07 12:51:01


On Aug 3, 2012, at 6:24 PM, Paul Kapinos wrote:

> testing our well-known example of the registered memory problem (see http://www.open-mpi.org/community/lists/users/2012/02/18565.php) on freshly-installed 1.6.1rc2, found out that "Fall back to send/receive semantics" did not work always it. However the behaviour has changed:
>
> 1.5.3. and older: MPI processes hang and block the IB interface(s) forever
>
> 1.6.1rc2: a) MPI processes run through (if the chunk size is less than 8Gb) with or without a warning; or
> b) MPI processes die (if the chunk size is more than 8Gb)

We talked about this mail today on our weekly teleconference.

That's odd. Looking at your output files, I see that they when trying to create a queue pair. Let me explain...

Our newest stop-gap scheme on the 1.6 branch is as follows:

- figure out how much physical RAM is on the machine
- take 85% of that number
- M = (85% of physical_memory / num_mpi_procs_on_machine)
- don't let any individual MPI process register more than M bytes of memory

This is a heuristic. The idea is that we leave 15% of memory available to the OS and other OpenFabrics services running on the machine (IPoIB, subnet management, ...etc.). However, there is a variable OMPI doesn't count -- the amount of registered memory consumed by the meta data consumed by a queue pair.

When you take into account the fact that OMPI creates queue pairs lazily (in an attempt to reduce registered memory consumption, which is fairly ironic here ;-) ), we could still run out of registered memory and then try to create a new QP later (e.g., the first time MPI process A sends to B). This QP could fail to be created if there is no more registered memory.

That's the type of error that I see in your log files (QP creation fail).

But with 15% of RAM left, we're greatly surprised to see this kind of error. Perhaps registering 8+GB buffers does something in the OpenFabrics registration system that we're unaware of (to make overall available registered memory deplete faster). Huh.

More below.

> Note that the same program which die in (b) run fine over IPoIB (-mca btl ^openib). However, the performance is very bad in this case... some 1100 sec. instead of about a minute.

Yep, that makes sense. IPoIB is quite inefficient.

> Reproducing: compile attached file and let it run on nodes with >=24GB with
> log_num_mtt : 20
> log_mtts_per_seg: 3
> (=32Gb, our default values):
> $ mpiexec ....<one proc per node> .... a.out 1080000000 1080000001

So I'm not 100% clear on what you mean here: when you set the OFED params to allow registration of more memory than you have physically, does the problem go away?

>From your log messages, the warning messages were from machines with nearly 100GB RAM but only 32GB register-able. But only one of those was the same as one that showed QP creation failures. So I'm not clear which was which.

Regardless: can you pump the MTT params up to allow registering all of physical memory on those machines, and see if you get any failures?

To be clear: we're trying to determine if we should spend more effort on making OMPI work properly in low-registered-memory-availabile scenarios, or whether we should just emphasize "go reset your MTT parameters to allow registering all of physical memory."

> Well, we know about the need to raise the values of one of these parameters, but I wanted to let you to know that your workaround for the problem is still not 100% perfect but only 99%.

Ok, good. Many thanks for your patience with all of this.

> P.S: A note about the informative warning:
> --------------------------------------------------------------------------
> WARNING: It appears that your OpenFabrics subsystem is configured to only
> allow registering part of your physical memory.
> ....
> Registerable memory: 32768 MiB
> Total memory: 98293 MiB
> --------------------------------------------------------------------------
> On node with 24 GB this warning did not came around, although the max. size of registered memory (32GB) is only 1.5x of RAM, but in
> http://www.open-mpi.org/faq/?category=openfabrics#ib-low-reg-mem
> at least the 2x RAM size is recommended.
>
> Should this warning not came out in all cases when registered memory < 2x RAM?

You're correct -- perhaps this is bad wording in the FAQ. As far as I understand it, it's only necessary to be able to register all of physical memory. Although Mellanox did recommend being able to register at least 2x physical memory.

...that being said, I have never gotten a clear explanation of what exactly those two parameters are. I.e., why you would adjust one and not the other, etc. Mellanox advised us (the OMPI developers) to adjust log_num_mtt, but I never found out why.

We'll continue to pester Mellanox to try to get a good answer as to why 2x is recommended. :-) If we get a good answer, we'll update the FAQ wording and/or the limits at which that warning is displayed.

-- 
Jeff Squyres
jsquyres_at_[hidden]
For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/