Galen Shipman wrote:
>You probably are not seeing overhead costs so much as you are seeing
>the difference between using send/recv for small messages, which Open
>MPI uses, and RDMA for small messages. If you are comparing against
>another implementation that uses RDMA for small messages then yes,
>will see lower latencies, but there are issues with using small
>RDMA. I have written a paper that addresses these issues which will
>presented at IPDPS.
I've been working for the MVAPICH project for around three years. Since
this thread is discussing MVAPICH, I thought I should post to this
thread. Galen's description of MVAPICH is not accurate. MVAPICH uses
RDMA for short message to deliver performance benefits to the
applications. However, it needs to be designed properly to handle
scalability while delivering best performance. Since MVAPICH-0.9.6
(released on 6th December, 2005), MVAPICH has been supporting a new mode
of operation which is called ADAPTIVE_RDMA_FAST_PATH (the basic
RDMA_FAST_PATH is also supported).
This new design uses RDMA for short message transfer in an intelligent
and adaptive manner. Using this mode, the memory allocation of MVAPICH
is no longer static. Instead its dynamic. Its an implementation of the
short message RDMA implementation for a limited set of peers (user
controllable) which Galen is suggesting. MVAPICH already supports this
feature. This also means that in the paper Galen mentions, the
comparison results in Figures 4 through 7 have to be re-evaluated to
make the paper and the results accurate.
Hope this helps.