Open MPI logo

Open MPI Development Mailing List Archives

  |   Home   |   Support   |   FAQ   |  

This web mail archive is frozen.

This page is part of a frozen web archive of this mailing list.

You can still navigate around this archive, but know that no new mails have been added to it since July of 2016.

Click here to be taken to the new web archives of this list; it includes all the mails that are in this frozen archive plus all new mails that have been sent to the list since it was migrated to the new archives.

Subject: Re: [OMPI devel] RFC: Add GPU Direct RDMA support to openib btl
From: Kenneth A. Lloyd (kenneth.lloyd_at_[hidden])
Date: 2013-10-08 19:23:18



From: devel [mailto:devel-bounces_at_[hidden]] On Behalf Of Rolf vandeVaart
Sent: Tuesday, October 08, 2013 3:05 PM
To: devel_at_[hidden]
Subject: [OMPI devel] RFC: Add GPU Direct RDMA support to openib btl


WHAT: Add GPU Direct RDMA support to openib btl

WHY: Better latency for small GPU message transfers

WHERE: Several files, see ticket for list

WHEN: Friday, October 18, 2013 COB

More detail:

This RFC looks to make use of GPU Direct RDMA support that is coming in the
future in Mellanox libraries. With GPU Direct RDMA, we can register GPU
memory with the ibv_reg_mr() calls. Therefore, we are simply piggy backing
on the large message RDMA support (RGET) that exists in the PML and openib
BTL. For best performance, we want to use the RGET protocol at small
messages and the switch to a pipeline protocol at larger messages.


To make use of this, we add some extra code paths that are followed when
moving GPU buffers. If we have the support compiled in, then when we
detect we have a GPU buffer, we use the RGET protocol even for small
messages. When the messages get larger, we switch to using the regular
pipeline protocol. There is some other support code that is added as well.
We add a flag to any GPU memory that is registered so we can check for
cuMemAlloc/cuMemFree/cuMemAlloc issues. Each GPU has a buffer ID associated
with it, so we can ensure that any registrations in the rcache are still


To view the changes, go to
and click on gdr.diff
f> .




This email message is for the sole use of the intended recipient(s) and may
contain confidential information. Any unauthorized review, use, disclosure
or distribution is prohibited. If you are not the intended recipient,
please contact the sender by reply email and destroy all copies of the
original message.