Open MPI logo

Open MPI Development Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Development mailing list

Subject: [OMPI devel] PRE-RFC: Adding RDMA support for GPU buffers within a node
From: Rolf vandeVaart (rvandevaart_at_[hidden])
Date: 2011-09-16 16:08:18


This is a pre-RFC of some changes I am hoping to bring into the trunk.
(I call this a pre-RFC as I have no timeout and I am not done with the code
yet.)

With some prior commits, I have added the ability to send GPU buffers directly.
This support consists of forcing the use of only the send/receive protocol when
a GPU buffer is detected and using special memcpy functions to move data into
and out of these buffers.

My next set of changes intends to improve the performance within a node when
sending GPU buffers. Essentially, we can do a RDMA remote get operations between
two GPU cards.

This code adds support for that. On the sending side, we use a very simple
mpool to get a memory handle. On the receiveing side, we use a mpool to register
and cache the memory handles. It is patterned after the rdma mpool.

I am interested in anyone's reaction to this code. Most of the changes
are contained in within code that is only used for CUDA, but there are
several changes that happen within the sm BTL and the ob1 PML. What was
kind of strange is that with these changes, we have the sm BTL that supports
RDMA operations for GPU buffers, but not for host buffers, and that made
for interesting code.

The current changes can be viewed at:
  https://bitbucket.org/rolfv/ompi-trunk-cuda-rdma-1

[rvandevaart_at_rvandevaart-dt ompi-trunk-interp-rdma]$ svn status
M opal/datatype/opal_convertor.h
M ompi/mca/btl/sm/btl_sm_frag.c
M ompi/mca/btl/sm/btl_sm.c
M ompi/mca/btl/sm/btl_sm_component.c
M ompi/mca/btl/sm/btl_sm_frag.h
M ompi/mca/btl/sm/btl_sm.h
M ompi/mca/btl/sm/Makefile.am
M ompi/mca/btl/btl.h
A ompi/mca/mpool/cuda
A ompi/mca/mpool/cuda/mpool_cuda_component.c
A ompi/mca/mpool/cuda/mpool_cuda_module.c
A ompi/mca/mpool/cuda/mpool_cuda.h
A ompi/mca/mpool/cuda/Makefile.am
A ompi/mca/mpool/rcuda
A ompi/mca/mpool/rcuda/mpool_rcuda_component.c
A ompi/mca/mpool/rcuda/Makefile.am
A ompi/mca/mpool/rcuda/mpool_rcuda_module.c
A ompi/mca/mpool/rcuda/mpool_rcuda.h
M ompi/mca/pml/ob1/pml_ob1_rdma.h
M ompi/mca/pml/ob1/pml_ob1_sendreq.c
M ompi/mca/pml/ob1/pml_ob1_sendreq.h
M ompi/mca/pml/ob1/pml_ob1_recvreq.c
M ompi/mca/pml/ob1/pml_ob1_rdma.c
M ompi/mca/common/cuda/common_cuda.c
M ompi/mca/common/cuda/common_cuda.h

-----------------------------------------------------------------------------------
This email message is for the sole use of the intended recipient(s) and may contain
confidential information. Any unauthorized review, use, disclosure or distribution
is prohibited. If you are not the intended recipient, please contact the sender by
reply email and destroy all copies of the original message.
-----------------------------------------------------------------------------------