Open MPI logo

Open MPI Development Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Development mailing list

From: Don Kerr (Don.Kerr_at_[hidden])
Date: 2007-07-13 09:07:57


Jeff Squyres wrote:

>On Jul 12, 2007, at 1:18 PM, Don Kerr wrote:
>
>
>
>>>- So if you want to simply eliminate the flow control, choose M high
>>>enough (or just a total number of receive buffers to post to the SRQ)
>>>that you won't ever run out of resources and you should see some
>>>speedup from lack of flow control. This obviously mainly helps apps
>>>with lots of small messages; it may not help in many other cases.
>>>
>>>
>>>
>>Is there any distinction by the size of the message. If the "M"
>>parameter is set high does the openib btl post this many recv buffers
>>for the SRQ on both QPs? Or are SRQs only created on one of the QPs?
>>
>>
>
>Keep in mind that the SRQs are only for send/receive messages, not
>RDMA messages.
>
>
That is obviously enough but isn't there a window for MPI messages that
are greater than the eager limit but less than where the rdma protocol
kicks in and fragments for this size message use fragments larger than
than the eager size.

Maybe this is where openib's high and low priority qp differ from udapl
which makes a choice of which endpoint to use based on the size of the
fragment. That is why I was curious if openib was using SRQs on both
queue pairs.

>Each receive buffer has a max size (the eager limit, IIRC). So if
>the message is larger than that, we'll fragment per the pipeline
>protocol, possibly subject to doing RDMA if the message is large
>enough, yadda yadda yadda. More specifically, the size of the buffer
>is not dependent upon an individual message that is being sent or
>received (since they're pre-posted -- we have no idea what the
>message sizes will be).
>
>As for whether the SRQ is on both QP's, this is a Galen/George/Gleb
>(G^3) question...
>
>
>