On Thu, 25 Jun 2009, Eugene Loh wrote:
> I spoke with Brian and Jeff about this earlier today. Presumably, up through
> 1.2, mca_btl_component_progress would poll and if it received a message
> fragment would return. Then, presumably in 1.3.0, behavior was changed to
> keep polling until the FIFO was empty. Brian said this was based on Terry's
> desire to keep latency as low as possible in benchmarks. Namely, reaching
> down into a progress call was a long code path. It would be better to pick
> up multiple messages, if available on the FIFO, and queue extras up in the
> unexpected queue. Then, a subsequent call could more efficiently find the
> anticipated message fragment.
> I don't see how the behavior would impact short-message pingpongs (the
> typical way to measure latency) one way or the other.
> I asked Terry, who struggled to remember the issue and pointed me at this
> thread: http://www.open-mpi.org/community/lists/devel/2008/06/4158.php .
> But that is related to an issue that's solved if one keeps polling as long as
> one gets ACKs (but returns as soon as a real message fragment is found).
> Can anyone shed some light on the history here? Why keep polling even when a
> message fragment has been found? The downside of polling too aggressively is
> that the unexpected queue can grow (without bounds).
> Brian's proposal is to set some variable that determines how many message
> fragments a single mca_btl_sm_component_progress call can drain from the FIFO
> before returning.
I checked, and 1.3.2 definitely drains all messages until the fifo is
empty. If we were to switch to drain until we receive a data message and
that fixes Terry's issue, that seems like a rational change and would not
require the fix I suggested. My assumption had been that we needed to
drain more than one data message per call to component_progress in order
to work around Terry's issue. If not, then let's go with the simple fix
and only drain one data message per enterance to component_progress (but
drain multiple acks if we have a bunch of acks and then a data message in
Unfortunately I have no more history than what Terry proposed, but it
looks like the changes were made around that time.