Open MPI logo

Open MPI Development Mailing List Archives

  |   Home   |   Support   |   FAQ   |  

This web mail archive is frozen.

This page is part of a frozen web archive of this mailing list.

You can still navigate around this archive, but know that no new mails have been added to it since July of 2016.

Click here to be taken to the new web archives of this list; it includes all the mails that are in this frozen archive plus all new mails that have been sent to the list since it was migrated to the new archives.

Subject: Re: [OMPI devel] [ofa-general] uDAPL EVD queue length issue
From: Steve Wise (swise_at_[hidden])
Date: 2007-12-05 11:08:29


Jon Mason wrote:
> On Tue, Dec 04, 2007 at 11:40:17AM -0800, Arlin Davis wrote:
>> Jon Mason wrote:
>>> While working on OMPI udapl btl, I have noticed some "interesting"
>>> behavior. OFA udapl wants the evd queues to be a power of 2 and
>>> then will subtract 1 for book keeping (ie, so that internal head and
>>> tail pointers never touch except when the ring is empty). OFA udapl
>>> will report the queue length as this number (and not the original
>>> size requested) when queried. This becomes interesting when a power
>>> of 2 is passed in and then queried. For example, a requested queue
>>> of length 256 will report a length of 255 when queried.
>> Something is not right. You should ALWAYS get at least what you request. On
>> my system with an mthca, a request of 256 gets you 511. It is the verbs
>> provider that is rounding up, not uDAPL.
>>
>> Here is my uDAPL debug output (DAPL_DBG_TYPE=0xffff) using dtest:
>>
>> cq_object_create: (0x519bb0,0x519d00)
>> dapls_ib_cq_alloc: evd 0x519bb0 cqlen=256
>> dapls_ib_cq_alloc: new_cq 0x519d60 cqlen=511
>>
>> This is before and after the ibv_create_cq call. uDAPL builds it's EVD
>> resources based on what is returned from this call.
>>
>> I modified dtest to double check the dat_evd_query and I get the same:
>>
>> 8962 dto_rcv_evd created 0x519e80
>> 8962 dto_req_evd QLEN - requested 256 and actual 511
>>
>> What OFED release and device are you using?
>
> I'm running OFED 1.2.5 and using Chelsio.
>
> The behavior of the iwch_create_cq in
> drivers/infiniband/hw/cxgb3/iwch_provider.c is to allocate the amount
> given (rounded to the power of 2). So this function will give 256 if
> 256 is requested, but uDAPL will consume one of those for book keeping
> and thus only have 255.
>
> For my clarification, the provider should take into account the
> bookkeeping of uDAPL and roundup to the next power of 2 when given a
> power of 2 size? I'm probably being thick, but why doesn't uDAPL
> increase the size requested by one before passing the request to the
> provider (or is this the documented behavior of the function and the
> provider should conform)?
>
> Thanks,
> Jon

 From the linux rdma verbs perspective, ibv_create_cq() will create a cq
that is >= the requested depth. The fact that mthca always bumps the
size up to the next power of 2 isn't something udapl can rely on.

Here's the crux: If creating a udapl evd of 256 results in a cq of 256
and the udapl returns a evd with size 255, then udapl is broken...

My 2 cents...

Stevo.