Brian W. Barrett wrote:
> All -
> Jeff, Eugene, and I had a long discussion this morning on the sm BTL
> flow management issues and came to a couple of conclusions.
> * Jeff, Eugene, and I are all convinced that Eugene's addition of
> polling the receive queue to drain acks when sends start backing up is
> required for deadlock avoidance.
> * We're also convinced that George's proposal, while a good idea in
> general, is not sufficient. The send path doesn't appear to
> sufficiently progress the btl to avoid the deadlocks we're seeing with
> the SM btl today. Therefore, while I still recommend sizing the fifo
> appropriately and limiting the freelist size, I think it's not
> sufficient to solve all problems.
> * Finally, it took an hour, but we did determine one of the major
> differences between 1.2.8 and 1.3.0 in terms of sm is how messages
> were pulled off the FIFO. In 1.2.8 (and all earlier versions), we
> return from btl_progress after a single message is received (ack or
> message) or the fifo was empty. In 1.3.0 (pre-srq work Eugene did),
> we changed to completely draining all queues before returning from
> btl_progress. This has led to a situation where a single call to
> btl_progress can make a large number of callbacks into the PML
> (900,000 times in one of Eugene's test case). The change was made to
> resolve an issue Terry was having with performance of a benchmark.
> We've decided that it would be adventageous to try something between
> the two points and drain X number of messages from the queue, then
> return, where X is 100 or so at most. This should cover the
> performance issues Terry saw, but still not cause the huge number of
> messages added to the unexpected queue with a single call to
> MPI_Recv. Since a recv that is matched on the unexpected queue
> doesn't result in a call to opal_progress, this should help balance
> the load a little bit better. Eugene's going to take a stab at
> implementing this short term.
I checked with Terry and we can't really recover the history here.
Perhaps draining ACKs is good enough. After the first message, we can
That's a one-line change. Just comment out line 432 ("goto
Problem is, that doesn't "fix" things. That is, my deadlock avoidance
stuff (hg workspace on milliways that I sent out a pointer to) seems to
be enough to, well, avoid deadlock, but unexpected-message queues are
still growing like mad I think. Even when sm progress returns after the
first message fragment is received. (X=1.) I think it's even true if
the max free-list size is capped at something small. I *think* (but am
too tired to "know") that the issue is we poll the FIFO often anyhow.
We have to for sends to reclaim fragments. We have to for receives, to
pull out messages of interest. Maybe things would be better if we had
one FIFO for in-coming fragments and another for returning fragments.
We could poll the latter only when we needed another fragment for sending.
But I'm under pressure to shift my attention to other activities. So, I
think I'm going to abandon this effort. The flow control problem seems
thorny. I can think of fixes as fast as I can identify flow-control
problems, but the rate of new flow-control problems just doesn't seem to
abate. Meanwhile, my unexpected-work queue grows unbounded. :^)
> I think the combination of Euegene's deadlock avoidance fix and the
> careful queue draining should make me comfortable enough to start
> another round of testing, but at least explains the bottom line issues.