Open MPI logo

FAQ:
Running CUDA-aware Open MPI

  |   Home   |   Support   |   FAQ   |   all just the FAQ

Table of contents:

  1. What kind of CUDA support exists in Open MPI?
  2. Which MPI APIs work with CUDA-aware?
  3. Which MPI APIs do NOT work with CUDA-aware?
  4. Can I tell at compile time or runtime whether I have CUDA-aware support?
  5. How do I limit how much CUDA IPC memory is held in the registration cache?


1. What kind of CUDA support exists in Open MPI?

Since Open MPI v1.7.0, there is support for sending and receiving CUDA device memory directly. Prior to this support, the programmer would first have to stage the data in host memory prior to making the MPI calls. Now, the Open MPI library will automatically detect that the pointer being passed in is a CUDA device memory pointer and do the right thing. This is referred to as CUDA-aware support.

The use of device pointers is supported in all of the send and receive APIs as well as the blocking collective APIs. Neither the nonblocking collective APIs nor the accumulate one-sided APIs are supported.

See this FAQ entry for more details on which APIs are supported.

Open MPI depends on various features of CUDA 4.0, so one needs to have at least the CUDA 4.0 driver and toolkit. The new features of interest are the Unified Virtual Addressing (UVA) so that all pointers within a program have unique addresses. In addition, there is a new API that allows one to determine if a pointer is a CUDA device pointer or host memory pointer. This API is used by the library to decide what needs to be done with each buffer. In addition, CUDA 4.1 also provides the ability to register host memory with the CUDA driver which can improve performance. CUDA 4.1 also added CUDA IPC support for fast communication between GPUs on the same node.

Note that derived datatypes, both contiguous and non-contiguous, are supported. However, the non-contiguous datatypes currently have high overhead because of the many calls to cuMemcpy to copy all the pieces of the buffer into the intermediate buffer.

CUDA-aware support is only available in the sm, smcuda, tcp, and openib BTLs. The smcuda BTL is an optimized version of the sm BTL that takes advantage of the CUDA IPC support for fast GPU transfers. Much of the other optimizations are built in to the openib BTL.

There is no CUDA-aware support in any of the MTLs.

Open MPI v1.7.0, Open MPI v1.7.1, Open MPI v1.7.2

  • Basic GPU direct support.
  • Support for CUDA IPC between GPUs on a node, but would get error if the GPUs did not support CUDA IPC.

Open MPI v1.7.3 New Features

  • Support for asynchronous copies of larger GPU buffers over the openib BTL.
  • Dynamically loads the libcuda.so library so you can configure with CUDA-aware support, but run on machines that do not have CUDA installed.

Open MPI v1.7.4 New Features

  • Removed synchronize point in CUDA IPC when running with CUDA 6.0 or later.
  • Utilizes GPU Direct RDMA if it is available. Requires CUDA 6.0 or later.
  • Dynamically enable CUDA IPC support between GPUs and back off to copy through host memory if it is not available.

Open MPI v1.8.0 - v1.8.4 New Features

  • Minor error handling fixes.
  • Better cleanup of CUDA resources

Open MPI v1.8.5 New Features

  • Improved on-node GPU to GPU transfers even when CUDA IPC is not supported between the two GPUs.
  • Properly handle Unified Memory. This is done by disabling CUDA IPC and GPU Direct RDMA optimizations on Unified Memory buffers.
  • Support for blocking reduction MPI APIs.

For best results, it is recommended that you use the latest version of Open MPI which as of this writing was Open MPI v1.10.1.

Additional Information about CUDA-aware support

Here are some relevant mca parameters to extract extra information if you are having issues. For Open MPI v1.7.3 and later, you can see if the library was built with CUDA-aware support.

1
2
shell$ ompi_info --parsable --all | grep mpi_built_with_cuda_support:value
mca:mpi:base:param:mpi_built_with_cuda_support:value:true

To get some extra information, there are some verbose flags. The opal_cuda_verbose has only one level of verbosity. (Works on all versions)

1
shell$ mpirun --mca opal_cuda_verbose 10 ...

This mpi_common_cuda_verbose flag provides additional information about CUDA-aware related activities. This can be set to a variety of different values. There is really no need to use these unless you have strange problems (works on all versions).

1
2
3
shell$ mpirun --mca mpi_common_cuda_verbose 10 ...
shell$ mpirun --mca mpi_common_cuda_verbose 20 ...
shell$ mpirun --mca mpi_common_cuda_verbose 100 ...

There are three new MCA parameters introduced with Open MPI v1.7.4 related to the use of CUDA IPC. By default, CUDA IPC is used where possible. But the user can now turn it off if they want.

1
shell$ mpirun --mca btl_smcuda_use_cuda_ipc 0 ...

In addition, it is assumed that CUDA IPC is possible when running on the same GPU and this is typically true. However, there is the ability to turn it off.

1
shell$ mpirun --mca btl_smcuda_use_cuda_ipc_same_gpu 0 ...

Lastly, to get some insight into whether CUDA IPC is being used, you can turn on some verbosity that shows whether CUDA IPC gets enabled between two GPUs.

1
shell$ mpirun --mca btl_smcuda_cuda_ipc_verbose 100 ...

GPU Direct RDMA Information

Open MPI v1.7.4 and later have added some support to take advantage of GPU Direct RDMA on Mellanox cards. All the details about Mellanox hardware as well as software needed to get things to work can be found at Mellanox web site. Note that to get GPU Direct RDMA support, you also need to configure your Open MPI library with CUDA 6.0.

To see if you have GPU Direct RDMA compiled into your library, you can check like this:

1
2
shell$ ompi_info --all | grep btl_openib_have_cuda_gdr
   MCA btl: informational "btl_openib_have_cuda_gdr" (current value: "true", data source: default, level: 4 tuner/basic, type: bool)

To see if your OFED stack has GPU Direct RDMA support, you can check like this:

1
2
shell$ ompi_info --all | grep btl_openib_have_driver_gdr
   MCA btl: informational "btl_openib_have_driver_gdr" (current value: "true", data source: default, level: 4 tuner/basic, type: bool)

To run with GPU Direct RDMA support, you have to enable it as it is off by default.

1
shell$ mpirun --mca btl_openib_want_cuda_gdr 1 ...

GPU Direct RDMA Implementation Details

With GPU Direct RDMA support selected, the eager protocol is unused. This is done to avoid the penalty of copying unexpected GPU messages into host memory. Instead, a rendezvous protocol is used where the sender and receiver both register their GPU buffers and make use of GPU Direct RDMA support to transfer the data. This is done for all messages that are less than 30,000 bytes in size. For larger messages, the openib BTL switches to using pipelined buffers as that has better performance at larger messages. So, by default, with GPU Direct RDMA enabled, the underlying protocol usage is like this:

1
2
0      < message size < 30,000      GPU Direct RDMA
30,000 < message size < infinity    Asynchronous copies through host memory

You can adjust the point where we switch to asynchronous copes with the --mca btl_openib_cuda_rdma_limit value. For example, if you want to increase the switchover point to 100,000 bytes, then set it like this.

1
shell$ mpirun --mca btl_openib_cuda_rdma_limit 100000 ...

By default, if we have GPU Direct RDMA, we use it for 1 byte messages on up to the btl_openib_cuda_rdma_limit value. However, you could use the eager protocol for the smallest messages by setting --mca btl_openib_cuda_eager_limit value. Note: The btl_openib_cuda_eager_limit value includes some overhead so you cannot just set it to the payload value. It has to be set to the payload plus the extra upper layer extra bytes. Currently, in Open MPI v1.7.4, this overhead is 44 bytes, so that has to be the minimum value. In the table below we are just referring to the size of the payload.

This table tries to show how the various runtime parameters affect what protocols are used in a GPU Direct RDMA.

Message Size Limits Protocol
0 < message size < btl_openib_cuda_eager_limit (default=0) eager protocol (not used by default)
btl_openib_cuda_eager_limit (default=0) < message size < btl_openib_cuda_rdma_limit (default=30,000) rendezvous protocol utilizing GPU Direct RDMA
btl_openib_cuda_rdma_limit (default=30,000) < message size < infinity pipelined transfers of size 128K through host memory

Performance Note The cost of registering the GPU memory with the Mellanox driver is expensive so it is best to reuse the same GPU buffer for communication.

NUMA Node Issues When running on a node that has multiple GPUs, you may want to select the GPU that is closest to the process you are running on. One way to do this is to make use of the hwloc library. Following is a code snippet that can be used in your application to select a GPU that is close. It will determine which CPU it is running on and then look for the closest GPU. There could be multiple GPUs that are the same distance away. This is dependent on having hwloc somewhere on your system.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
/**
 * Test program to show the use of hwloc to select the GPU closest to the CPU
 * that the MPI program is running on.  Note that this works even without
 * any libpciacces or libpci support as it keys of the NVIDIA vendor ID.
 * There may be other ways to implement this but this is one way.
 * January 10, 2014
 */
#include <assert.h>
#include <stdio.h>
#include "cuda.h"
#include "mpi.h"
#include "hwloc.h"
 
#define ABORT_ON_ERROR(func)                          \
  { CUresult res;                                     \
    res = func;                                       \
    if (CUDA_SUCCESS != res) {                        \
        printf("%s returned error=%d\n", #func, res); \
        abort();                                      \
    }                                                 \
  }
static hwloc_topology_t topology = NULL;
static int gpuIndex = 0;
static hwloc_obj_t gpus[16] = {0};
 
/**
 * This function searches for all the GPUs that are hanging off a NUMA
 * node.  It walks through each of the PCI devices and looks for ones
 * with the NVIDIA vendor ID.  It then stores them into an array.
 * Note that there can be more than one GPU on the NUMA node.
 */
 
static void find_gpus(hwloc_topology_t topology, hwloc_obj_t parent, hwloc_obj_t child) {
    hwloc_obj_t pcidev;
    pcidev = hwloc_get_next_child(topology, parent, child);
    if (NULL == pcidev) {
        return;
    } else if (0 != pcidev->arity) {
        /* This device has children so need to look recursively at them */
        find_gpus(topology, pcidev, NULL);
        find_gpus(topology, parent, pcidev);
    } else {
        if (pcidev->attr->pcidev.vendor_id == 0x10de) {
            gpus[gpuIndex++] = pcidev;
        }
        find_gpus(topology, parent, pcidev);
    }
}
int main(int argc, char *argv[])
{
    int rank, retval, length;
    char procname[MPI_MAX_PROCESSOR_NAME+1];
    const unsigned long flags = HWLOC_TOPOLOGY_FLAG_IO_DEVICES | HWLOC_TOPOLOGY_FLAG_IO_BRIDGES;
    hwloc_cpuset_t newset;
    hwloc_obj_t node, bridge;
    char pciBusId[16];
    CUdevice dev;
    char devName[256];
 
    MPI_Init(&argc, &argv);
    MPI_Comm_rank(MPI_COMM_WORLD, &rank);
    if (MPI_SUCCESS != MPI_Get_processor_name(procname, &length)) {
        strcpy(procname, "unknown");
    }
 
    /* Now decide which GPU to pick.  This requires hwloc to work properly.
     * We first see which CPU we are bound to, then try and find a GPU nearby.
     */
    retval = hwloc_topology_init(&topology);
    assert(retval == 0);
    retval = hwloc_topology_set_flags(topology, flags);
    assert(retval == 0);
    retval = hwloc_topology_load(topology);
    assert(retval == 0);
    newset = hwloc_bitmap_alloc();
    retval = hwloc_get_last_cpu_location(topology, newset, 0);
    assert(retval == 0);
 
    /* Get the object that contains the cpuset */
    node = hwloc_get_first_largest_obj_inside_cpuset(topology, newset);
 
    /* Climb up from that object until we find the HWLOC_OBJ_NODE */
    while (node->type != HWLOC_OBJ_NODE) {
        node = node->parent;
    }
 
    /* Now look for the HWLOC_OBJ_BRIDGE.  All PCI busses hanging off the
     * node will have one of these */
    bridge = hwloc_get_next_child(topology, node, NULL);
    while (bridge->type != HWLOC_OBJ_BRIDGE) {
        bridge = hwloc_get_next_child(topology, node, bridge);
    }
 
    /* Now find all the GPUs on this NUMA node and put them into an array */
    find_gpus(topology, bridge, NULL);
 
    ABORT_ON_ERROR(cuInit(0));
    /* Now select the first GPU that we find */
    if (gpus[0] == 0) {
        printf("No GPU found\n");
        exit(1);
    } else {
        sprintf(pciBusId, "%.2x:%.2x:%.2x.%x", gpus[0]->attr->pcidev.domain, gpus[0]->attr->pcidev.bus,
        gpus[0]->attr->pcidev.dev, gpus[0]->attr->pcidev.func);
        ABORT_ON_ERROR(cuDeviceGetByPCIBusId(&dev, pciBusId));
        ABORT_ON_ERROR(cuDeviceGetName(devName, 256, dev));
        printf("rank=%d (%s): Selected GPU=%s, name=%s\n", rank, procname, pciBusId, devName);
    }
 
    MPI_Finalize();
    return 0;
}

See this FAQ entry for detals on how to configure the CUDA support into the library.


2. Which MPI APIs work with CUDA-aware?

MPI API Support Added In Version
MPI_Send, MPI_Bsend, MPI_Ssend, MPI_Rsend, MPI_Isend, MPI_Ibsend, MPI_Issend, MPI_Irsend, MPI_Send_init, MPI_Bsend_init, MPI_Ssend_init, MPI_Rsend_init, MPI_Recv, MPI_Irecv, MPI_Recv_init, MPI_Sendrecv, MPI_Bcast, MPI_Gather, MPI_Gatherv, MPI_Allgather, MPI_Allgatherv, MPI_Alltoall, MPI_Alltoallv, MPI_Alltoallw, MPI_Scatter, MPI_Scatterv Open MPI v1.7.0
MPI_Win_create, MPI_Put, MPI_Get Open MPI v1.8.0
MPI_Reduce, MPI_Allreduce, MPI_Scan, MPI_Exscan, MPI_Reduce_scatter, MPI_Reduce_scatter_block Open MPI v1.8.5


3. Which MPI APIs do NOT work with CUDA-aware?

MPI API Expected Support
MPI_Accumulate, MPI_Rget, MPI_Rput, MPI_Get_Accumulate, MPI_Fetch_and_op, MPI_Compare_and_swap Future
MPI_Iallgather, MPI_Iallgatherv, MPI_Iallreduce, MPI_Ialltoall, MPI_Iialltoallv, MPI_Ialltoallw, MPI_Ibcast, MPI_Iexscan Future


4. Can I tell at compile time or runtime whether I have CUDA-aware support?

New with Open MPI v2.0.0, we have added a compile time check and a runtime check. You can use whichever is the most convenient for your program. To access them, you need to include mpi-ext.h. Note that mpi-ext.h has been around for several releases so you can just add it to your include list. The following program shows an example of using the CUDA-aware macro and runtime check.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
/*
 * Program that shows the use of CUDA-aware macro and runtime check.
 * Requires Open MPI v2.0.0 or later.
 */
#include <stdio.h>
#include "mpi.h"
 
#ifdef
#include "mpi-ext.h" /* Needed for CUDA-aware check */
#endif
 
int main(int argc, char *argv[])
{
    printf("Compile time check:\n");
#if defined(MPIX_CUDA_AWARE_SUPPORT) && MPIX_CUDA_AWARE_SUPPORT
    printf("This MPI library has CUDA-aware support.\n", MPIX_CUDA_AWARE_SUPPORT);
#elif defined(MPIX_CUDA_AWARE_SUPPORT) && !MPIX_CUDA_AWARE_SUPPORT
    printf("This MPI library does not have CUDA-aware support.\n");
#else
    printf("This MPI library cannot determine if there is CUDA-aware support.\n");
#endif /* MPIX_CUDA_AWARE_SUPPORT */
 
    printf("Run time check:\n");
#if defined(MPIX_CUDA_AWARE_SUPPORT)
    if (1 == MPIX_Query_cuda_support()) {
        printf("This MPI library has CUDA-aware support.\n");
    } else {
        printf("This MPI library does not have CUDA-aware support.\n");
    }
#else /* !defined(MPIX_CUDA_AWARE_SUPPORT) */
    printf("This MPI library cannot determine if there is CUDA-aware support.\n");
#endif /* MPIX_CUDA_AWARE_SUPPORT */
 
    return 0;
}
 


5. How do I limit how much CUDA IPC memory is held in the registration cache?

As mentioned earlier, the Open MPI library will make use of CUDA IPC support where possible to move the GPU data quickly between GPUs that are on the same node and same PCI root complex. The library holds on to registrations even after the data transfer is complete as it is expensive to make some of the CUDA IPC registration calls. If you want to limit how much memory is registered, you can use the mpool_rgpusm_rcache_size_limit MCA parameter. For example, this sets the limit to 1000000 bytes.

1
shell$ --mca mpool_rgpusm_rcache_size_limit 1000000

When the cache reaches this size, it will kick out the least recently used until it can fit the new registration in.

In Open MPI 1.10.2 and later, there also is the ability to have the cache empty itself out when the limit is reached. To do this, just set the MCA parameter as shown in this example.

1
shell$ --mca mpool_rgpusm_rcache_empty_cache 1