I also believe that for iWARP and RoCE, the RDMA CM will be chosen automatically, and UD CM will be automatically chosen for IB.
[Josh] If you want to run OMPI over RoCE on Mellanox hardware, you must explicitly choose rdmacm with -mca btl openib,sm,self -mca btl_openib_cpc_include rdmacm - that's THE way to tell OMPI to run over RoCE and create RoCE QPs.
So after all that: I think you shouldn't need to specify the connection manager MCA parameter at all; the openib BTL should choose the Right one for you.
[Josh] Nyet. See above.
[Josh] And, yes, RDMACM has scalability issues.