Open MPI logo

Open MPI Development Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Development mailing list

Subject: Re: [OMPI devel] why mx_forget in mca_btl_mx_prepare_dst?
From: George Bosilca (bosilca_at_[hidden])
Date: 2009-10-21 14:06:33


On Oct 21, 2009, at 13:42 , Scott Atchley wrote:

> On Oct 21, 2009, at 1:25 PM, George Bosilca wrote:
>
>> Brice,
>>
>> Because MX doesn't provide a real RMA protocol, we created a fake
>> one on top of point-to-point. The two peers have to agree on a
>> unique tag, then the receiver posts it before the sender starts the
>> send. However, as this is integrated with the real RMA protocol,
>> where only one side knows about the completion of the RMA
>> operation, we still exchange the ACK at the end. Therefore, the
>> receiver doesn't need to know when the receive is completed, as it
>> will get an ACK from the sender. At least this was the original idea.
>>
>> But I can see how this might fails if the short ACK from the sender
>> manage to pass the RMA operation on the wire. I was under the
>> impression (based on the fact that MX respect the ordering) that
>> the mx_send will trigger the completion only when all data is on
>> the wire/nic memory so I supposed there is _absolutely_ no way for
>> the ACK to bypass the last RMA fragments and to reach the receiver
>> before the recv is really completed. If my supposition is not
>> correct, then we should remove the mx_forget and make sure the that
>> before we mark a fragment as completed we got both completions (the
>> one from mx_recv and the remote one).
>
> George,
>
> When is the ACK sent? After the "PUT" completion returns (via mx_test
> (), etc) or simply after calling mx_isend() for the "PUT" but before
> the completion?

The ACK is sent by the PML layer. If I'm not mistaken, it is sent when
the completion callback is triggered, which should happen only when
the MX BTL detect the completion of the mx_isend (using the mx_test).
Therefore, I think the ACK is sent in response to the completion of
the mx_isend.

   george.

> If the former, the ACK cannot pass the data. If the latter, it is
> easily possible especially if there is a lot of contention (and thus
> a lot of route dispersion).
>
> MX only guarantees order of matching (two identical tags will match
> in order), not order of completion.
>
> Scott
> _______________________________________________
> devel mailing list
> devel_at_[hidden]
> http://www.open-mpi.org/mailman/listinfo.cgi/devel