> > So far, the best latency I got from ompi is 5.24 us, and the best I
> > got from mvapich is 3.15.
> > I am perfectly ready to accept that ompi scales better and that may be
> > more important (except to the marketing dept :-) ), but I do not
> > understand your explanation based on small-message RDMA. Either I
> > missunderstood something badly (my best guess), or the 2 us are lost to
> > something else than an RDMA-size tradeoff.
> Again this is small message RDMA with polling versus send/receive
> semantics, we will be adding small message RDMA and should have
> performance equal to that of mvapich for small messages, but it is only
> relevant for a small working set of peers / micro benchmarks.
Thanks a lot. I was being fooled by various levels of size thresholds in
the mvapich code. It was indeed doing rdma for small messages. After
turning that off, I get numbers comparable to yours. Well, mvapich still
beats ompi by a hair on my configuration. 5.11 vs. 5.25 but that's in
the near-irrelevant range compared to other benefits.
>From an adoption perspective, though, the ability to shine in
micro-benchmarks is important, even if it means using an ad-hoc tuning.
There is some justification for it after all. There are small clusters
out there (many more than big ones, in fact) so taking maximum advantage
of a small scale is relevant.
When do you plan on having the small-msg rdma option available ?
Jean-Christophe Hugly <jice_at_[hidden]>