Open MPI logo

Open MPI Development Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Development mailing list

Subject: Re: [OMPI devel] [ofa-general] uDAPL EVD queue length issue
From: Arlin Davis (ardavis_at_[hidden])
Date: 2007-12-04 14:40:17


Jon Mason wrote:
> While working on OMPI udapl btl, I have noticed some "interesting"
> behavior. OFA udapl wants the evd queues to be a power of 2 and
> then will subtract 1 for book keeping (ie, so that internal head and
> tail pointers never touch except when the ring is empty). OFA udapl
> will report the queue length as this number (and not the original
> size requested) when queried. This becomes interesting when a power
> of 2 is passed in and then queried. For example, a requested queue
> of length 256 will report a length of 255 when queried.

Something is not right. You should ALWAYS get at least what you request.
On my system with an mthca, a request of 256 gets you 511. It is the
verbs provider that is rounding up, not uDAPL.

Here is my uDAPL debug output (DAPL_DBG_TYPE=0xffff) using dtest:

  cq_object_create: (0x519bb0,0x519d00)
dapls_ib_cq_alloc: evd 0x519bb0 cqlen=256
dapls_ib_cq_alloc: new_cq 0x519d60 cqlen=511

This is before and after the ibv_create_cq call. uDAPL builds it's EVD
resources based on what is returned from this call.

I modified dtest to double check the dat_evd_query and I get the same:

8962 dto_rcv_evd created 0x519e80
8962 dto_req_evd QLEN - requested 256 and actual 511

What OFED release and device are you using?

-arlin