Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: Re: [OMPI users] Program hangs when using OpenMPI and CUDA
From: Brice Goglin (Brice.Goglin_at_[hidden])
Date: 2011-06-05 02:20:54

Le 05/06/2011 00:15, Fengguang Song a écrit :
> Hi,
> I'm confronting a problem when using OpenMPI 1.5.1 on a GPU cluster. My program uses MPI to exchange data
> between nodes, and uses cudaMemcpyAsync to exchange data between Host and GPU devices within a node.
> When the MPI message size is less than 1MB, everything works fine. However, when the message size
> is > 1MB, the program hangs (i.e., an MPI send never reaches its destination based on my trace).
> The issue may be related to locked-memory contention between OpenMPI and CUDA.
> Does anyone have the experience to solve the problem? Which MCA parameters should I tune to increase
> the message size to be > 1MB (to avoid the program hang)? Any help would be appreciated.
> Thanks,
> Fengguang


I may have seen the same problem when testing GPU direct. Do you use the
same host buffer for copying from/to GPU and for sending/receiving on
the network ? If so, you need a GPUDirect enabled kernel and mellanox
drivers, but it only helps before 1MB.

You can work around the problem with one of the following solution:
* add --mca btl_openib_flags 304 to force OMPI to always send/recv
through an intermediate (internal buffer), but it'll decrease
performance before 1MB too
* use different host buffers for the GPU and the network and manually
copy between them

I never got any reply from NVIDIA/Mellanox/here when I reported this
problem with GPUDirect and messages larger than 1MB.