So when openMPI is compile with OFED, and run on a Infiniband/RoCE devices, I would use the mpi would simply direct to ofed to do point to point calls in the ofed way?
More specifically: all things being equal, you don't care which is used. You just want your message to get to the receiver/target as fast as possible. One of the main ideas of MPI is to hide those kinds of details from the user. I.e., you call MPI_SEND. A miracle occurs. The message is received on the other side.
True. Its just that I am digging into the OFED source code and the ompi source code,and trying to understand the way these two interact..