Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: Re: [OMPI users] XRC vs SRQ vs PRQ
From: Brock Palen (brockp_at_[hidden])
Date: 2013-01-22 16:06:32


On Jan 22, 2013, at 2:53 PM, Shamis, Pavel wrote:

>>
>> Switching to SRQ and some guess of queue values selected appears to let the code run.
>> S,4096,128:S,12288,128:S,65536,12
>>
>> Two questions,
>>
>> This is a ConnectX fabric, should I switch them to XRC queues? And should I use the same queue size/count? That a safe assumption?
>> X,4096,128:X,12288,128:X,65536,12
>
> Yeah, I would use the same values as a starting point.

Thanks, the users full resolution job got further with shared queues, we are going to do a test with XRC queues of the same count. But he keeps getting OpenMPI out of memory/reg fail messages.

>
>>
>>
>> When should I use one queue type over the other?
>
> Generally speaking XRC transport has much better scalability that RC.

Ok so if we are useing shared queues on ConnectX gear default to XRC, will do.

>
>
>>
>> Is there a way to get stat feedback on the use of your shared queues (SRQ or XRC) ?
>>
>> Example, using code 'not from here' and would like to know, "hey you are always running out of your queue of size X" Or " your queue of size Y is never used"
>>
>> We are kinda blind for a lot of our applications :-)
>
> Right now we don't have such hooks in openib BTL.
> It is not very difficult to add some code that will report stat for QP utilization.
>
> In you other email you mentioned MXM. I would recommend to try both XRC and MXM and see which one performance better. On relatively small system I would guess
> XRC will perform better, on large system MXM should demonstrate better performance. But again, it all depends on your application.
>
> - Pasha
>
>
> _______________________________________________
> users mailing list
> users_at_[hidden]
> http://www.open-mpi.org/mailman/listinfo.cgi/users