Open MPI logo

Open MPI Development Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Development mailing list

Subject: [OMPI devel] reduce problem/#1791
From: Jeff Squyres (jsquyres_at_[hidden])
Date: 2009-02-17 12:54:41


Per the call today, I ran the reduce-hang.c test on a single x86 4-
core xeon node with both 1.2.8 and 1.2.9. I see the same behavior
with both:

- none of the processes hang; they all keep iterating over MPI_REDUCE
- the iteration numbers sent to stdout are more-or-less in sync
- memory usage (via top) is fairly steady
   - ...except when MPI_COMM_WORLD rank 0 memory usage periodically
jumps up significantly

I do not believe that this is a 1.2.9 regression; see below.

For example, I've had the test running for 40+ minutes right now;
here's the current output from top:

-----
   PID USER PR NI %CPU TIME+ %MEM VIRT RES SHR S COMMAND
  7332 jsquyres 25 0 100 43:56.79 36.1 33.8g 1.3g 67m R reduce-
hang
  7333 jsquyres 25 0 100 43:54.10 0.7 202m 26m 18m R reduce-
hang
  7335 jsquyres 25 0 100 43:56.94 1.0 203m 35m 26m R reduce-
hang
  7334 jsquyres 25 0 100 43:56.81 1.0 203m 36m 27m R reduce-
hang
-----

33.8GB virtual size, 1.3GB resident. Obviously, this number started
off much lower than that; it took several "jumps" to get to that
size. It's now more-or-less stable at these sizes; I haven't seen any
jumps recently.

This smells quite a bit like MCW rank 0 simply got busy and "fell
behind" for some [temporary] reason. It therefore accumulated a whole
pile of unexpected messages, thereby pushing memory utilization up.
Those unexpected messages eventually got consumed, but the memory is
placed on a freelist and not released.

This is also congruent with Brad's observation that he's much more
likely to see this behavior if he's on a machine with SMT enabled --
where it's much more likely that MCW rank 0 can "fall behind", and
possibly never catch up.

So my guess is that this is the same old unbounded unexpected message
issue that we've known about for a while.

-- 
Jeff Squyres
Cisco Systems