Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: Re: [OMPI users] anybody tried OMPI with gpudirect?
From: Rolf vandeVaart (rvandevaart_at_[hidden])
Date: 2011-02-28 13:49:38


For the GPU Direct to work with Infiniband, you need to get some updated OFED bits from your Infiniband vendor.

In terms of checking the driver updates, you can do a grep on the string get_driver_pages in the file/proc/kallsyms. If it is there, then the Linux kernel is updated correctly.

The GPU Direct functioning should be independent of the MPI you are using.

Rolf

-----Original Message-----
From: users-bounces_at_[hidden] [mailto:users-bounces_at_[hidden]] On Behalf Of Brice Goglin
Sent: Monday, February 28, 2011 11:42 AM
To: users_at_[hidden]
Subject: Re: [OMPI users] anybody tried OMPI with gpudirect?

Le 28/02/2011 17:30, Rolf vandeVaart a écrit :
> Hi Brice:
> Yes, I have tired OMPI 1.5 with gpudirect and it worked for me. You definitely need the patch or you will see the behavior just as you described, a hang. One thing you could try is disabling the large message RDMA in OMPI and see if that works. That can be done by adjusting the openib BTL flags.
>
> -- mca btl_openib_flags 304
>
> Rolf
>

Thanks Rolf. Adding this mca parameter worked-around the hang indeed.
The kernel is supposed to be properly patched for gpudirect. Are you
aware of anything else we might need to make this work? Do we need to
rebuild some OFED kernel modules for instance?

Also, is there any reliable/easy way to check if gpudirect works in our
kernel ? (we had to manually fix the gpudirect patch for SLES11).

Brice

_______________________________________________
users mailing list
users_at_[hidden]
http://www.open-mpi.org/mailman/listinfo.cgi/users
-----------------------------------------------------------------------------------
This email message is for the sole use of the intended recipient(s) and may contain
confidential information. Any unauthorized review, use, disclosure or distribution
is prohibited. If you are not the intended recipient, please contact the sender by
reply email and destroy all copies of the original message.
-----------------------------------------------------------------------------------