Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: [OMPI users] gpudirect p2p?
From: Chris Cooper (rallycoops_at_[hidden])
Date: 2011-10-14 01:27:58


Hi,

Are the recent peer to peer capabilities of cuda leveraged by Open MPI
when eg you're running a rank per gpu on the one workstation?

It seems in my testing that I only get in the order of about 1GB/s as
per http://www.open-mpi.org/community/lists/users/2011/03/15823.php,
whereas nvidia's simpleP2P test indicates ~6 GB/s.

Also, I ran into a problem just trying to test. It seems you have to
do cudaSetDevice/cuCtxCreate with the appropriate gpu id which I was
wanting to derive from the rank. You don't however know the rank
until after MPI_Init() and you need to initialize cuda before. Not
sure if there's a standard way to do it? I have a workaround atm.

Thanks,
Chris