>> For Heroic latencies on IB we would need to use small message RDMA and
>> poll each peers dedicated memory region for completion.
> Well, I tried to play around with the eager_limit, min_rdma, etc. I did
> not see the latency of messages of a given size be lowered by changing
> the tresholds to make them rdma'd. Rather the opposite (which was my
> initial expectation, actually). May be I just misunderstood the whole
> set of tunables. My understanding was that messages under the eager
> limit would never be rdma'd by definition, and that the others would or
> would not be, depending on the min_rdma_size.
The current code always uses send/recv semantics for short/eager messages.
As Galen indicated, we will be adding support for short message rdma write,
but this is generally not scalable, and would only be viable for a small
working set of peers.