Open MPI logo

Open MPI Development Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Development mailing list

Subject: Re: [OMPI devel] [ofa-general] uDAPL EVD queue length issue
From: Steve Wise (swise_at_[hidden])
Date: 2007-12-05 11:08:29


Jon Mason wrote:
> On Tue, Dec 04, 2007 at 11:40:17AM -0800, Arlin Davis wrote:
>> Jon Mason wrote:
>>> While working on OMPI udapl btl, I have noticed some "interesting"
>>> behavior. OFA udapl wants the evd queues to be a power of 2 and
>>> then will subtract 1 for book keeping (ie, so that internal head and
>>> tail pointers never touch except when the ring is empty). OFA udapl
>>> will report the queue length as this number (and not the original
>>> size requested) when queried. This becomes interesting when a power
>>> of 2 is passed in and then queried. For example, a requested queue
>>> of length 256 will report a length of 255 when queried.
>> Something is not right. You should ALWAYS get at least what you request. On
>> my system with an mthca, a request of 256 gets you 511. It is the verbs
>> provider that is rounding up, not uDAPL.
>>
>> Here is my uDAPL debug output (DAPL_DBG_TYPE=0xffff) using dtest:
>>
>> cq_object_create: (0x519bb0,0x519d00)
>> dapls_ib_cq_alloc: evd 0x519bb0 cqlen=256
>> dapls_ib_cq_alloc: new_cq 0x519d60 cqlen=511
>>
>> This is before and after the ibv_create_cq call. uDAPL builds it's EVD
>> resources based on what is returned from this call.
>>
>> I modified dtest to double check the dat_evd_query and I get the same:
>>
>> 8962 dto_rcv_evd created 0x519e80
>> 8962 dto_req_evd QLEN - requested 256 and actual 511
>>
>> What OFED release and device are you using?
>
> I'm running OFED 1.2.5 and using Chelsio.
>
> The behavior of the iwch_create_cq in
> drivers/infiniband/hw/cxgb3/iwch_provider.c is to allocate the amount
> given (rounded to the power of 2). So this function will give 256 if
> 256 is requested, but uDAPL will consume one of those for book keeping
> and thus only have 255.
>
> For my clarification, the provider should take into account the
> bookkeeping of uDAPL and roundup to the next power of 2 when given a
> power of 2 size? I'm probably being thick, but why doesn't uDAPL
> increase the size requested by one before passing the request to the
> provider (or is this the documented behavior of the function and the
> provider should conform)?
>
> Thanks,
> Jon

 From the linux rdma verbs perspective, ibv_create_cq() will create a cq
that is >= the requested depth. The fact that mthca always bumps the
size up to the next power of 2 isn't something udapl can rely on.

Here's the crux: If creating a udapl evd of 256 results in a cq of 256
and the udapl returns a evd with size 255, then udapl is broken...

My 2 cents...

Stevo.