Open MPI logo

Open MPI Development Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Development mailing list

Subject: Re: [OMPI devel] reduce problem/#1791
From: Jeff Squyres (jsquyres_at_[hidden])
Date: 2009-02-17 14:41:19


Brad replied with more information here:

     https://svn.open-mpi.org/trac/ompi/ticket/1791#comment:3

On Feb 17, 2009, at 12:54 PM, Jeff Squyres wrote:

> Per the call today, I ran the reduce-hang.c test on a single x86 4-
> core xeon node with both 1.2.8 and 1.2.9. I see the same behavior
> with both:
>
> - none of the processes hang; they all keep iterating over MPI_REDUCE
> - the iteration numbers sent to stdout are more-or-less in sync
> - memory usage (via top) is fairly steady
> - ...except when MPI_COMM_WORLD rank 0 memory usage periodically
> jumps up significantly
>
> I do not believe that this is a 1.2.9 regression; see below.
>
> For example, I've had the test running for 40+ minutes right now;
> here's the current output from top:
>
> -----
> PID USER PR NI %CPU TIME+ %MEM VIRT RES SHR S COMMAND
> 7332 jsquyres 25 0 100 43:56.79 36.1 33.8g 1.3g 67m R reduce-
> hang
> 7333 jsquyres 25 0 100 43:54.10 0.7 202m 26m 18m R reduce-
> hang
> 7335 jsquyres 25 0 100 43:56.94 1.0 203m 35m 26m R reduce-
> hang
> 7334 jsquyres 25 0 100 43:56.81 1.0 203m 36m 27m R reduce-
> hang
> -----
>
> 33.8GB virtual size, 1.3GB resident. Obviously, this number started
> off much lower than that; it took several "jumps" to get to that
> size. It's now more-or-less stable at these sizes; I haven't seen
> any jumps recently.
>
> This smells quite a bit like MCW rank 0 simply got busy and "fell
> behind" for some [temporary] reason. It therefore accumulated a
> whole pile of unexpected messages, thereby pushing memory
> utilization up. Those unexpected messages eventually got consumed,
> but the memory is placed on a freelist and not released.
>
> This is also congruent with Brad's observation that he's much more
> likely to see this behavior if he's on a machine with SMT enabled --
> where it's much more likely that MCW rank 0 can "fall behind", and
> possibly never catch up.
>
> So my guess is that this is the same old unbounded unexpected
> message issue that we've known about for a while.
>
> --
> Jeff Squyres
> Cisco Systems
>
> _______________________________________________
> devel mailing list
> devel_at_[hidden]
> http://www.open-mpi.org/mailman/listinfo.cgi/devel

-- 
Jeff Squyres
Cisco Systems