Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: Re: [OMPI users] MXM vs OpenIB
From: Shamis, Pavel (shamisp_at_[hidden])
Date: 2013-01-23 11:08:14


>>> You sound like our vendors, "what is your app"
>>
>> ;-) I used to be one.
>>
>> Ideally OMPI should do the switch between MXM/RC/XRC internally in the transport layer. Unfortunately,
>> we don't have such smart selection logic. Hopefully IB vendors will fix some day.
>
> I actually looked in the openib-hca.ini (working from memory) to try and find what the default queues were, and I actually couldn't figure it out. The ConnectX entry doesn't have a default, and the 'default default' also doesn't have an entry.
>
> I need to dig into ompi_info, got distracted by an intel compiler bug, ADD for admin/user support folks.

"default default" QP configuration is tuned for Mellanox devices.
Therefore the openib-hca.ini file doesn't have a special configuration for Connect-X.
The proper way to check default configuration is ompi_info utility.

>
>>
>>>
>>> Note most of our users run just fine with the standard Peer-Peer queues, default out the box OpenMPI.
>>
>> The P2P queue is fine, but most like using XRC your users will observe better performance. This is not just scalability.
>
> Cool thanks for all the input, I wonder why peer-to-peer is the default, I know XRC requires hardware support,

There is a historical reason behind this. OpenFabrics decided not to include XRC transport in default distribution. The XRC feature was only available as a part of
the Mellanox OFED distribution. I think, recently OFA community decided to include XRC, so we actually should consider to enable XRC by default.

-Pasha