Thanks Pasha for these details.
On Mon, 17 May 2010, Pavel Shamis (Pasha) wrote:
>> blocking is the receive queues, because they are created during MPI_Init,
>> so in a way, they are the "basic fare" of MPI.
> BTW SRQ resources are also allocated on demand. We start with very small SRQ
> and it is increased on SRQ limit event.
Ok. Understood. So maybe the increased memory is only due to CQs.
>> The XRC protocol seems to create shared receive queues, which is a good
>> thing. However, comparing memory used by an "X" queue versus and "S" queue,
>> we can see a large difference. Digging a bit into the code, we found some
> So, do you see that X consumes more that S ? This is really odd.
Yes, but that's what we see. At least after MPI_Init.
>> strange things, like the completion queue size not being the same as "S"
>> queues (the patch below would fix it, but the root of the problem may be
>> Is anyone able to comment on this ?
> The fix looks ok, please submit it to trunk.
I don't have an account to do this, so I'll let maintainers push it into
> BTW do you want to prepare the patch for send queue size factor ? It should
> be quite simple.
Maybe we can do this. However, we are a little playing with parameters and
code without really knowing the deep consequences of what we do.
Therefore, I would feel more confortable if someone who knows much on the
openib btl confirms it's not breaking everything.