Open MPI logo

Open MPI Development Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Development mailing list

Subject: Re: [OMPI devel] Infiniband memory usage with XRC
From: Pavel Shamis (Pasha) (pashash_at_[hidden])
Date: 2010-05-23 11:02:18

~ 2300 KB - is it difference per machine or per MPI process ?
In OMPI XRC mode we allocate some additional resources that may consume
some memory (the hash table), but even so ~2M sounds too much for me.
When I will have time I will try to calculate the "resonable" difference.


Sylvain Jeaugey wrote:
> On Mon, 17 May 2010, Pavel Shamis (Pasha) wrote:
>> Sylvain Jeaugey wrote:
>>>>> The XRC protocol seems to create shared receive queues, which is a
>>>>> good thing. However, comparing memory used by an "X" queue versus
>>>>> and "S" queue, we can see a large difference. Digging a bit into
>>>>> the code, we found some
>>>> So, do you see that X consumes more that S ? This is really odd.
>>> Yes, but that's what we see. At least after MPI_Init.
>> What is the difference (in Kb)?
> At 32 nodes x 32 cores (1024 MPI processes), I get a difference of
> ~2300 KB in favor of "S,65536,16,4,1" versus "X,65536,16,4,1".
> The proposed patch doesn't seem to solve the problem however, there's
> still something that's taking more memory than expected.
> Sylvain