Jon Mason wrote:
> While working on OMPI udapl btl, I have noticed some "interesting"
> behavior. OFA udapl wants the evd queues to be a power of 2 and
> then will subtract 1 for book keeping (ie, so that internal head and
> tail pointers never touch except when the ring is empty). OFA udapl
> will report the queue length as this number (and not the original
> size requested) when queried. This becomes interesting when a power
> of 2 is passed in and then queried. For example, a requested queue
> of length 256 will report a length of 255 when queried.
Something is not right. You should ALWAYS get at least what you request.
On my system with an mthca, a request of 256 gets you 511. It is the
verbs provider that is rounding up, not uDAPL.
Here is my uDAPL debug output (DAPL_DBG_TYPE=0xffff) using dtest:
dapls_ib_cq_alloc: evd 0x519bb0 cqlen=256
dapls_ib_cq_alloc: new_cq 0x519d60 cqlen=511
This is before and after the ibv_create_cq call. uDAPL builds it's EVD
resources based on what is returned from this call.
I modified dtest to double check the dat_evd_query and I get the same:
8962 dto_rcv_evd created 0x519e80
8962 dto_req_evd QLEN - requested 256 and actual 511
What OFED release and device are you using?