Open MPI logo

Open MPI Development Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Development mailing list

Subject: Re: [OMPI devel] RFC: Add GPU Direct RDMA support to openib btl
From: Kenneth A. Lloyd (kenneth.lloyd_at_[hidden])
Date: 2013-10-08 19:23:18


+1

 

From: devel [mailto:devel-bounces_at_[hidden]] On Behalf Of Rolf vandeVaart
Sent: Tuesday, October 08, 2013 3:05 PM
To: devel_at_[hidden]
Subject: [OMPI devel] RFC: Add GPU Direct RDMA support to openib btl

 

WHAT: Add GPU Direct RDMA support to openib btl

WHY: Better latency for small GPU message transfers

WHERE: Several files, see ticket for list

WHEN: Friday, October 18, 2013 COB

More detail:

This RFC looks to make use of GPU Direct RDMA support that is coming in the
future in Mellanox libraries. With GPU Direct RDMA, we can register GPU
memory with the ibv_reg_mr() calls. Therefore, we are simply piggy backing
on the large message RDMA support (RGET) that exists in the PML and openib
BTL. For best performance, we want to use the RGET protocol at small
messages and the switch to a pipeline protocol at larger messages.

 

To make use of this, we add some extra code paths that are followed when
moving GPU buffers. If we have the support compiled in, then when we
detect we have a GPU buffer, we use the RGET protocol even for small
messages. When the messages get larger, we switch to using the regular
pipeline protocol. There is some other support code that is added as well.
We add a flag to any GPU memory that is registered so we can check for
cuMemAlloc/cuMemFree/cuMemAlloc issues. Each GPU has a buffer ID associated
with it, so we can ensure that any registrations in the rcache are still
valid.

 

To view the changes, go to https://svn.open-mpi.org/trac/ompi/ticket/3836
and click on gdr.diff
<https://svn.open-mpi.org/trac/ompi/ticket/3836%20and%20click%20on%20gdr.dif
f> .

 

 

  _____

This email message is for the sole use of the intended recipient(s) and may
contain confidential information. Any unauthorized review, use, disclosure
or distribution is prohibited. If you are not the intended recipient,
please contact the sender by reply email and destroy all copies of the
original message.

  _____