Open MPI logo

Open MPI Development Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Development mailing list

Subject: Re: [OMPI devel] [PATCH] openib: clean-up connect to allow for new cm's
From: Pavel Shamis (Pasha) (pasha_at_[hidden])
Date: 2007-12-13 03:49:45


Gleb Natapov wrote:
> On Wed, Dec 12, 2007 at 04:08:31PM +0200, Pavel Shamis (Pasha) wrote:
>
>> Gleb Natapov wrote:
>>
>>> On Wed, Dec 12, 2007 at 03:37:26PM +0200, Pavel Shamis (Pasha) wrote:
>>>
>>>
>>>> Gleb Natapov wrote:
>>>>
>>>>
>>>>> On Tue, Dec 11, 2007 at 08:16:07PM -0500, Jeff Squyres wrote:
>>>>>
>>>>>
>>>>>> Isn't there a better way somehow? Perhaps we should have "select"
>>>>>> call *all* the functions and accept back a priority. The one with the
>>>>>> highest priority then wins. This is quite similar to much of the
>>>>>> other selection logic in OMPI.
>>>>>>
>>>>>> Sidenote: Keep in mind that there are some changes coming to select
>>>>>> CPCs on a per-endpoint basis (I can't look up the trac ticket right
>>>>>> now...). This makes things a little complicated -- do we need
>>>>>> btl_openib_cpc_include and btl_openib_cpc_exclude MCA params to
>>>>>> include/exclude CPCs (because you might need more than one CPC in a
>>>>>> single job)? That wouldn't be hard to do.
>>>>>>
>>>>>> But then what to do about if someone sets to use some XRC QPs and
>>>>>> selects to use OOB or RDMA CM? How do we catch this and print an
>>>>>> error? It doesn't seem right to put the "if num_xrc_qps>0" check in
>>>>>> every CPC. What happens if you try to make an XRC QP when not using
>>>>>> xoob? Where is the error detected and what kind of error message do
>>>>>> we print?
>>>>>>
>>>>>>
>>>>>>
>>>>> In my opinion "X" notation for QP specification should be removed. I
>>>>> didn't want this to prevent XRC merging so I haven't raced this point.
>>>>> It is enough to have two types of QPs "P" - SW credit management "S" -
>>>>> HW credit management.
>>>>>
>>>> How will you decide witch QP type to use ? (SRQ or XRC)
>>>>
>>>>
>>>>
>>> If both sides support XOOB and priority of XOOB is higher then all other
>>> CPC
>>> then create XRC, otherwise use regular RC.
>>>
>>>
>> If some body have connectX hca but he want to use SRQ and not XRC ?
>>
> This will be the default. (prio of OOB will be bigger than of XOOB), but
> if uses will want to use XRC it will increase XOOB priority by
> specifying MCA parameter.
>
>
>> I guess anyway we will be need some additional parameter that will allow
>> enable/disable XRC, correct ? (So why just not leave the X qp type ?)
>>
> Because we want to support mixed setups and create XRC between nodes that
> support it and RC between all other nodes.
>
Ok, sounds reasonable for me.
Just need make sure that the parameters name will be user friendly.
Some thing like --mca enable-xrc that will cause to XOOB priority be
highest (and not something like --mca xoob 10 :-))

Pasha