Open MPI logo

Open MPI Development Mailing List Archives

  |   Home   |   Support   |   FAQ   |  

This web mail archive is frozen.

This page is part of a frozen web archive of this mailing list.

You can still navigate around this archive, but know that no new mails have been added to it since July of 2016.

Click here to be taken to the new web archives of this list; it includes all the mails that are in this frozen archive plus all new mails that have been sent to the list since it was migrated to the new archives.

Subject: Re: [OMPI devel] Threaded progress for CPCs
From: Steve Wise (swise_at_[hidden])
Date: 2008-05-19 16:44:04


Jeff Squyres wrote:
> On May 19, 2008, at 3:40 PM, Jon Mason wrote:
>
>
>>>> iWARP needs preposted recv buffers (or it will drop the
>>>> connection). So
>>>> this isn't a good option.
>>>>
>>> I was talking about SRQ only. You said above that iwarp does
>>> retransmit for SRQ.
>>> openib BTL relies on HW retransmit when using SRQ, so if iwarp
>>> doesn't do it
>>> reliably enough it can not be used with SRQ anyway.
>>>
>> How iWARP adapters behave with respect to SRQ retransmit is 100% HW
>> dependent.
>>
>
> It was my understanding that it's at least the same as how TCP handles
> a dropped packet. The HW may do better than that.
>
>
>> The HW can queue some of the receives internally or use the HW TCP
>> stack to have
>> it retransmit. Of course, this is a BAD thing to do. The SRQ "low-
>> water marker"
>> event is the best way to handle these cases.
>>
>
>
> I disagree. I even think that the IB-retry-forever approach is bad.
> Here's why:
>
> 1. Posting more at low watermark can lead to DoS-like behavior when
> you have a fast sender and a slow receiver. This is exactly the
> resource-exhaustion kind of behavior that a high quality MPI
> implementation is supposed to avoid -- we really should to throttle
> the sender somehow.
>
> 2. Resending ad infinitum simply eats up more bandwidth and takes away
> network resources (e.g., switch resources) that other, legitimate
> traffic. Particularly if the receiver doesn't dip into the MPI layer
> for many hours. So yes, it *works*, but it's definitely sub-optimal.
>
>
The SRQ low water mark is simply an API method to allow applications to
try and never hit the "we're totally out recv bufs" problem. That's a
tool that I think is needed for srq users no matter what flow control
method you use to try and avoid jeff's #1 item above.

And if you don't like RNR retry/TCP retrans approach, which is bad for
reason #2 (and because TCP will eventually give up and reset the
connection), then I think there needs to be some OMPI layer protocol to
stop senders that are abusing the SRQ pool for whatever reason (too fast
of a sender, sleeping beauty receiver never entering OMPI layer, whtaever).

my 1/2 cent...

Steve.