Jeff Squyres wrote:
> On Nov 29, 2010, at 3:51 AM, Doron Shoham wrote:
>
>
>> If only the PUT flag is set and/or the btl supports only PUT method then the sender will allocate a rendezvous header and will not eager send any data. The receiver will schedule rdma PUT(s) of the entire message.
>> It is done in mca_pml_ob1_recv_request_schedule_once()
>> (ompi/mca/pml/ob1/pml_ob1_recvreq.c:683).
>> We can limit the size passing to mca_bml_base_prepare_dst() to be minimum between btl.max_message_size supported by the HCA and the actual message size.
>> The will result a fragmentation of the RDMA write messages.
>>
>
> I would think that we should set btl.max_message_size during init to be the minimum of the MCA param and the max supported by the HCA, right? Then there's no need for this min() in the critical path.
>
> Additionally, the message must be smaller than the max message size of *both* HCAs, right? So it might be necessary to add the max message size into the openib BTL modex data so that you can use it in mca_bml_base_prepare_dst() (or whatever -- been a long time since I've mucked around in there...) to compute the min between the two peers.
>
> So you might still need a min, but for a different reason than what you originally mentioned.
>
It is my mistake - the btl.max_message_size is a different parameter. It
is more like software limitation rather then hardware limitation from
the HCA.
I don't think that in RDMA flow it has any meaning.
Can you please explain a bit more about the openib BTL modex?
>
>> The bigger problem is when using the GET flow.
>> In this flow the receiver allocate one big buffer to receive the message with RDMA read in one chunk.
>> There is no fragmentation mechanism in this flow which make it harder to solve this issue
>>
>
> Doh. I'm afraid I don't know why this was done this way originally...
>
>
>> Reading the max message size supported by the HCA can be done by using verbs.
>>
>> The second approach is to use RDMA direct only if the message size is smaller than the max message size supported by the HCA.
>>
>> Here is where the long message protocol is chosen:
>> ompi/mca/pml/ob1/pml_ob1_sendreq.h line 382.
>>
>> We could use the second approach until a fragmentation mechanism will be added to the RDMA direct GET flow.
>>
>
> Are you suggesting that pml_ob1_sendreq.h:382 compare the message length to the btl.max_message_size and choose RDMA direct vs. RDMA pipelined? If so, that might be sufficient.
>
> But what to do about the peer's max message size?
>
>
I thought of a different approach:
Instead of limiting the passing to the mca_bml_base_prepare_dst(), we
can limit the size in mca_btl_openib_prepare_dst().
I believe this is better solution because it only effects the internal
behavior of the openib btl.
In mca_btl_openib_prepare_dst() we have access to both max_msg_sz (local
and endpoint).
This will fix the PUT flow.
For the GET flow, maybe we should check in
mca_pml_ob1_send_request_start_rdma() -
if the message size is larger then the minimum between both endpoints'
max_msg_sz force it use the PUT flow.
The problem is that I'm not sure how to do it without an *ugly hack*.
We need to to access the openib btl parameters from the
mca_pml_ob1_send_request_start_rdma().
The second options it to do it from pml_ob1_sendreq.h:382, but then
again, it requires access to the openib btl parameters...
Any thoughts?
Thanks,
Doron
|
![[Flag]](../../../../../community/mirrors/flag-us.gif){kind=small}