Open MPI logo

Open MPI Development Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Development mailing list

Subject: Re: [OMPI devel] [ofa-general] uDAPL EVD queue length issue
From: Steve Wise (swise_at_[hidden])
Date: 2007-12-06 07:52:08


Arlin Davis wrote:
>>>
>>> I'm running OFED 1.2.5 and using Chelsio.
>>>
>> From the linux rdma verbs perspective, ibv_create_cq() will create a
>> cq that is >= the requested depth. The fact that mthca always bumps
>> the size up to the next power of 2 isn't something udapl can rely on.
>
> It doesn't.
>
> uDAPL passes the users requested qlen directly to the verbs
> ibv_create_cq call (dapl_openib_cq.c) and then uses the returned qlen to
> allocate EVD's (dapl_evd_util.c) and a ring buffer
> (dapl_ring_buffer_util.c) for managing the free and pending events.
>
> The EVD's are allocated based on returned qlen from verbs and the
> ring_buffer is allocated based on qlen, next power of 2 minus 1. Unless
> I am missing something, I don't see how we can get less then what is
> requested.
>
>>
>> Here's the crux: If creating a udapl evd of 256 results in a cq of
>> 256 and the udapl returns a evd with size 255, then udapl is broken...
>
> Yes, I agree. But I don't see how this is happening.
>
> Here is output from my dtest when requesting 256 and verbs returning the
> same qlen. You can see before and after verbs we get 256, the rbufs are
> 511, and the query EVD call returns 256 to the user.
>
> cq_object_create: (0x519bb0,0x519d00)
> dapls_ib_cq_alloc: evd 0x519bb0 cqlen=256
> dapls_ib_cq_alloc: new_cq 0x519d60 cqlen=256
> setup_async_cb: ia 0x518270 type 2 hdl 0x519bb0 cb 0x2a957c8e70 ctx
> 0x519bb0
> >>> rbuf_alloc: size 256 rsize 511
> >>> rbuf_alloc: size 256 rsize 511
> dapl_evd_create () returns 0x0
>
> 9920 dto_req_evd QLEN - requested 256 and actual 256
> 9920 Create events done
>
> Can you turn up DAPL debug(DAPL_DBG_TYPE=0xffff) so I can see what is
> happening?
>
> Thanks,
>
> -arlin
>
>
>
>

Sorry for the noise guys. Looks like this is a cxgb3 bug.

Steve.