George Bosilca wrote:
> On Jun 23, 2009, at 11:04 , Eugene Loh wrote:
>> The sm BTL used to have two mechanisms for dealing with congested
>> FIFOs. One was to grow the FIFOs. Another was to queue pending
>> sends locally (on the sender's side). I think the grow-FIFO
>> mechanism was typically invoked and the pending-send mechanism used
>> only under extreme circumstances (no more memory).
>> With the sm makeover of 1.3.2, we dropped the ability to grow
>> FIFOs. The code added complexity and there seemed to be no need to
>> have two mechanisms to deal with congested FIFOs. In ticket 1944,
>> however, we see that repeated collectives can produce hangs, and
>> this seems to be due to the pending-send code not adequately dealing
>> with congested FIFOs.
>> Today, when a process tries to write to a remote FIFO and fails, it
>> queues the write as a pending send. The only condition under which
>> it retries pending sends is when it gets a fragment back from a
>> remote process.
>> I think the logic must have been that the FIFO got congested because
>> we issued too many sends. Getting a fragment back indicates that
>> the remote process has made progress digesting those sends. In
>> ticket 1944, we see that a FIFO can also get congested from too many
>> returning fragments. Further, with shared FIFOs, a FIFO could
>> become congested due to the activity of a third-party process.
>> In sum, getting a fragment back from a remote process is a poor
>> indicator that it's time to retry pending sends.
>> Maybe the real way to know when to retry pending sends is just to
>> check if there's room on the FIFO.
> Why this is different than "getting a fragment back"?
I'm not sure I understand your question.
Say we have two processes, A and B. Each one has a receive queue/FIFO
that can be written by its peer. Let's say A sends lots of messages to
B. B keeps on returning fragments to A. So, although we're saying that
A sends lots of messages to B, it is A's in-bound queue that fills up.
Kind of counterintuitive. Anyhow, B keeps getting more fragments to
return to A. Since A's queue is full, what this means is that B adds
these fragments to its (B's) own pending-send list.
So, now the question is when B should retry items on its pending-send
list. Presumably, it should retry when there is room on A's
queue/FIFO. But OMPI (to date) has B retry *only* when B itself gets a
fragment back. What's the logic? I assume the logic was that A's queue
was filled with fragments that B had sent, so getting a fragment back
would be an indication of A's queue opening up.
Why is this a poor indication? (I'm assuming this is what your question
was.) Two possible reasons:
1) A's queue might have been filled with fragments that B was returning
to A. So, B would get no acknowledgements back from A that progress was
being made depleting the queue.
2) (New with OMPI 1.3.2, now that we have shared queues): A's queue
might have been filled with activity from third party processes.
In either case, the only way B now knows whether there is room on A's
queue is... to check the queue if there's room! Nothing is coming back
from A to indicate that the queue is being drained.
> As far as I remember the code, when we get a fragment back we add it
> back in the LIFO, and therefore it become the next available fragment
> for a send.
Yes, indeed, but I don't understand how this is relevent. The LIFOs
(the private free lists where processes maintain unused fragments) don't
really enter this discussion.
>> So, I'll try modifying MCA_BTL_SM_FIFO_WRITE. It'll start by
>> checking if there are pending sends. If so, it'll retry them before
>> performing the requested write. This should also help preserve
>> ordering a little better. I'm guessing this will not hurt our
>> message latency in any meaningful way, but I'll check this out.
>> Meanwhile, I wanted to check in with y'all for any guidance you
>> might have.