That is discussed on here:

http://forums.nvidia.com/index.php?showtopic=227854

Maybe that could be your issue too.

From: users-bounces@open-mpi.org [mailto:users-bounces@open-mpi.org] On Behalf Of Rohan Deshpande
Sent: Tuesday, May 01, 2012 4:04 AM
To: Open MPI Users
Subject: [OMPI users] OpenMPI and CUDA on cluster

Hi,

I am trying to execute OpenMPI and CUDA code on a cluster. The code works fine on single machine but when I try to execute it on cluster I get error:

error while loading shared libraries: libcudart.so.4: cannot open shared object file: No such file or directory

I checked my PATH and LD_PATH and it looks ok. I have a .bashrc file which contains following entries -

export PATH=$PATH:/usr/local/lib/:/usr/local/lib/openmpi:/usr/local/ cuda/bin
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib:/usr/local/ lib/openmpi/:/usr/local/cuda/lib/:

All the machines haves same installation of CUDA and OpenMPI.

Can anyone help me with this.

This problem is really annoying.

Thanks.