Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: Re: [OMPI users] RoCE (IBoE) & OpenMPI
From: Doron Shoham (doron.ompi_at_[hidden])
Date: 2011-03-06 08:52:07


ibv_rc_pingpong runs over verbs and not RDMA_CM.
As far as I understood, RoCE requires the use of the RDMA CM (in open mpi),
and it is not possible to set explicitly the SL in RDMA_CM.
We should check if RDMA_CM is must for RoCE.

Thanks,
Doron

On Wed, Mar 2, 2011 at 8:01 PM, Jeff Squyres <jsquyres_at_[hidden]> wrote:

> Mellanox...?
>
> On Mar 2, 2011, at 12:51 PM, Michael Shuey wrote:
>
> > Honestly, I don't know - I haven't looked into the source. OFED 1.5.2
> > has a version of ibv_rc_pingpong that's been modified to work with
> > RoCE; you can pass the gid_index and SL as command-line arguments.
> > I'm not sure how that's handled at the IB layer, but the source may be
> > a good place to start.
> >
> > --
> > Mike Shuey
> >
> >
> >
> > On Tue, Mar 1, 2011 at 9:14 AM, Jeff Squyres <jsquyres_at_[hidden]> wrote:
> >> I thought you mentioned in a prior email that you had gotten one or two
> other OFED sample applications to work properly. How are they setting the
> SL? Are they not using the RDMA CM?
> >>
> >>
> >> On Mar 1, 2011, at 7:35 AM, Michael Shuey wrote:
> >>
> >>> So, since RoCE has no SM, and setting an SL is required to get
> >>> lossless ethernet on Cisco switches (and possibly others), does this
> >>> mean that RoCE will never work correctly with OpenMPI on Cisco
> >>> hardware?
> >>>
> >>> --
> >>> Mike Shuey
> >>>
> >>>
> >>>
> >>> On Tue, Mar 1, 2011 at 3:42 AM, Doron Shoham <dorons_at_[hidden]>
> wrote:
> >>>> Hi,
> >>>>
> >>>> Regarding to using a specific SL with RDMA CM, I've checked in the
> code and it seems that RDMA_CM uses the SL from the SA.
> >>>> So if you want to configure a specific SL, you need to do it via the
> SM.
> >>>>
> >>>> Doron
> >>>>
> >>>> -----Original Message-----
> >>>> From: Jeff Squyres [mailto:jsquyres_at_[hidden]]
> >>>> Sent: Thursday, February 24, 2011 3:45 PM
> >>>> To: Michael Shuey
> >>>> Cc: Open MPI Users , Mike Dubman
> >>>> Subject: Re: [OMPI users] RoCE (IBoE) & OpenMPI
> >>>>
> >>>> On Feb 24, 2011, at 8:00 AM, Michael Shuey wrote:
> >>>>
> >>>>> Late yesterday I did have a chance to test the patch Jeff provided
> >>>>> (against 1.4.3 - testing 1.5.x is on the docket for today). While it
> >>>>> works, in that I can specify a gid_index,
> >>>>
> >>>> Great! I'll commit that to the trunk and start the process of moving
> it to the v1.5.x series (I know you haven't tested it yet, but it's
> essentially the same patch, just slightly adjusted for each of the 3
> branches).
> >>>>
> >>>>> it doesn't do everything
> >>>>> required - my traffic won't match a lossless CoS on the ethernet
> >>>>> switch. Specifying a GID is only half of it; I really need to also
> >>>>> specify a service level.
> >>>>
> >>>> RoCE requires the use of the RDMA CM (I think?), and I didn't think
> there was a way to request a specific SL via the RDMA CM...? (I could
> certainly be wrong here)
> >>>>
> >>>> I think Mellanox will need to follow up with these questions...
> >>>>
> >>>> --
> >>>> Jeff Squyres
> >>>> jsquyres_at_[hidden]
> >>>> For corporate legal information go to:
> >>>> http://www.cisco.com/web/about/doing_business/legal/cri/
> >>>>
> >>>>
> >>>> _______________________________________________
> >>>> users mailing list
> >>>> users_at_[hidden]
> >>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
> >>>>
> >>>
> >>> _______________________________________________
> >>> users mailing list
> >>> users_at_[hidden]
> >>> http://www.open-mpi.org/mailman/listinfo.cgi/users
> >>
> >>
> >> --
> >> Jeff Squyres
> >> jsquyres_at_[hidden]
> >> For corporate legal information go to:
> >> http://www.cisco.com/web/about/doing_business/legal/cri/
> >>
> >>
> >> _______________________________________________
> >> users mailing list
> >> users_at_[hidden]
> >> http://www.open-mpi.org/mailman/listinfo.cgi/users
> >>
>
>
> --
> Jeff Squyres
> jsquyres_at_[hidden]
> For corporate legal information go to:
> http://www.cisco.com/web/about/doing_business/legal/cri/
>
>
> _______________________________________________
> users mailing list
> users_at_[hidden]
> http://www.open-mpi.org/mailman/listinfo.cgi/users
>