On Nov 14, 2013, at 12:21 PM, Barrett, Brian W <bwbarre_at_[hidden]> wrote:
> On 11/14/13 1:13 PM, "Joshua Ladd" <joshual_at_[hidden]> wrote:
>> Let me try to summarize my understanding of the situation:
>> 1. Ralph made the OOB asynchronous.
>> 2. OOB cpcs don't work as a result of 1, and are thereby "deprecated",
>> meaning: won't fix.
>> 3. Pasha moved the openib/connect to common/ofacm but excluded the rdmacm
>> in that move. Never changed openib to use ofacm/common.
>> 4. UDCM is "functional" in the trunk, still sitting in openib/connect.
>> But no one is entirely sure if it really works which is why it was
>> disabled in 1.7. Nathan - is there a design doc you can share on this
>> beyond the comments in the code?
>> 5. In order to satisfy the "grand plan":
>> a. UDCM still needs to be moved to common/ofacm.
>> b. OpenIB still needs to be changed to use common/ofacm.
>> c. RDMACM still needs to migrate to common/ofacm.
>> d. XRC support needs to be added to UDCM and put into
>> 6. The "grand plan" being: move the BTLs into Opal - hence the need to
>> scuttle the OOB cpcs thereby justifying the deprecation and not fixing
>> cpcs after #1.
>> So, that's a quick roundup of how we ended up here (as I understand it.)
>> What needs to be done is:
> That's my understanding as well.
>> 1. Somebody needs to certify/review/ that what Nathan has done is sound.
>> From my perspective, this is a BIG change and needs a comprehensive
>> architecture review. We've been using it in the trunk, and we've been
>> testing it under MTT for some time - but have not deployed or tested at
>> large-scale out in the field. Would be nice to see something on paper in
>> terms of a design doc.
>> 2. Somebody then needs to move UDCM into common/ofacm.
>> 3. Somebody needs to change openib to use common/ofacm cpcs instead of
>> openib/connect cpcs.
>> 4. Somebody needs to move RDMACM into common/ofacm and make sure RoCEE
>> 5. Somebody needs to add XRC support to UDCM - whatever that might mean.
>> Given Nathan added UDCM back in 2011 and nobody is really sure it's ready
>> for prime-time, and given Pasha's comments regarding the difference in
>> state machine requirements between the two connection schemes, this
>> doesn't seem like a trivial task.
>> Given Nathan's comments a second ago about ORNL not supporting the IB
>> Offload component, it barely makes sense to keep common/ofacm. And it
>> sounds like the two cpcs presently contained therein are now unusable.
>> All of this work is a result of the Grand Plan to move the BTLs into the
>> Opal layer - which I have no idea what the motive is (I was not involved
>> with OMPI when this was decided or debated.)
>> Basically, without these five changes OpenIB is dead in 1.7.4 and beyond
>> for RC, XRC, and RoCEE. These are blockers to 1.7.4 and I don't believe
>> that the onus falls squarely on Mellanox to fix these. These were
>> community decisions and, as such, it must be a community effort to
>> resolve. We are happy to lend a hand, but we are not fixing all of this
> I think that the 5 steps above sound correct and I agree that 1) this
> means 1.7.4 is on hold until we fix this and 2) that Mellanox shouldn't be
> the only one to fix this for 1.7.4, given the amount of work involved.
> Ralph, what, specifically, broke about the oob/xoob cpc mechanisms by
> making the oob asynchronous?
Hard for me to say as I don't really have access to an IB machine any more. Odin is my sole reference point, and someone has had that fully locked up for more than a week (and I can't complain as I am totally a guest there). Even then, I can only test on a few nodes.
I have no objection to helping, but we need someone who cares about IB and has access to such a machine to take the lead. Otherwise, we're just spinning our wheels.
As for the work issue: note that this has been "under development" now for more than a year. We've talked at length about how "somebody" needs to fix the openib/ofacm issue, but everyone keeps pushing it down the road as "not mine". Like I said, I can help - but (a) my boss couldn't care less about this issue, and (b) I have no way to test the results.
> That is, 1-5 are a huge amount of work; have
> we done the analysis to say that updating the oob / xoob cpc to work with
> the new oob is actually more work than doing 1-5? Obviously, there's long
> term plans that make oob/xoob problematic. But those aren't 1.7 / 1.8
> plans. Unfortunately, the cpcs were always out of my area of interest, so
> I'm flying a bit more blind than I'd like here.
> Brian W. Barrett
> Scalable System Software Group
> Sandia National Laboratories
> devel mailing list