On Sat, Aug 11, 2007 at 09:55:18AM -0700, Jeff Squyres wrote:
> With Mellanox's new HCA (ConnectX), extremely low latencies are
> possible for short messages between two MPI processes. Currently,
> OMPI's latency is around 1.9us while all other MPI's (HP MPI, Intel
> MPI, MVAPICH, etc.) are around 1.4us. A big reason for this
> difference is that, at least with MVAPICH, they are doing wire
> protocol header caching where the openib BTL does not. Specifically:
> - Mellanox tested MVAPICH with the header caching; latency was around
> - Mellanox tested MVAPICH without the header caching; latency was
> around 1.9us
As far as I remember Mellanox results and according to our testing
difference between MVAPICH with header caching and OMPI is 0.2-0.3us.
Not 0.5us. And MVAPICH without header caching is actually worse then
OMPI for small messages.
> Given that OMPI is the lone outlier around 1.9us, I think we have no
> choice except to implement the header caching and/or examine our
> header to see if we can shrink it. Mellanox has volunteered to
> implement header caching in the openib btl.
I think we have a chose. Not implement header caching, but just change the
osu_latency benchmark to send each message with different tag :)
I am not against header caching per se, but if it will complicate code
even a little bit I don't think we should implemented it just to benefit one
fabricated benchmark (AFAIR before header caching was implemented in
MVAPICH mpi_latency actually sent messages with different tags).
Also there is really nothing to cache in openib BTL. Openin BTL header is 4
bytes long. The caching will have to be done in OB1 and there it will
affect every other interconnect.
> Any objections? We can discuss what approaches we want to take
> (there's going to be some complications because of the PML driver,
> etc.); perhaps in the Tuesday Mellanox teleconf...?
My main objection is that the only reason you propose to do this is some
bogus benchmark? Is there any other reason to implement header caching?
I also hope you don't propose to break layering and somehow cache PML headers