Open MPI logo

Open MPI Development Mailing List Archives

  |   Home   |   Support   |   FAQ   |  

This web mail archive is frozen.

This page is part of a frozen web archive of this mailing list.

You can still navigate around this archive, but know that no new mails have been added to it since July of 2016.

Click here to be taken to the new web archives of this list; it includes all the mails that are in this frozen archive plus all new mails that have been sent to the list since it was migrated to the new archives.

Subject: Re: [OMPI devel] FDR: bad patch?
From: Yevgeny Kliteynik (kliteyn_at_[hidden])
Date: 2012-05-10 09:08:35


On 10-May-12 4:01 PM, Jeff Squyres wrote:
> Looks ok to me.
>
> FWIW, I'd put EDR in, if you know the values (even though there won't be any hardware for a while). That'll just give this code a longer shelf life / future-proof-ness...

Great, I'll add EDR too.
There's actually a patch pending to be accepted in kernel
that adds enums instead of all these values. But for now
we only have these numbers.

-- YK
 
>
> On May 10, 2012, at 8:57 AM, Yevgeny Kliteynik wrote:
>
>> On 10-May-12 3:42 PM, Yevgeny Kliteynik wrote:
>>>
>>> Here are the ActiveSpeed values:
>>>
>>> 1: SDR - 2.5 Gb/s * (8/10 coding)
>>> 2: DDR - 5 Gb/s * (8/10 coding)
>>> 4: QDR - 10 Gb/s * (8/10 coding)
>>> 8: FDR10 - 10 Gb/s * (64/66 coding)
>>> 16: FDR - 14 Gbps * (64/66 coding)
>>> 32: EDR - 25 Gbs * (64/66 coding)
>>>
>>> EDR is not out yet, so we can ignore it for the sake of this patch.
>>> It is also true that *most* of IB cables are 4x.
>>> There are 12x clusters out there (like TACC's Ranger), and sometimes
>>> there are 1x links (mostly because the cable is not perfect or too long).
>>>
>>> So cases of 4 and 8 are not the same for us, because MPI needs actual
>>> *data* rate instead of *bit* rate, which is different in these cases.
>>>
>>> I'll send a patch for review in a minute...
>>>
>>
>> How about this patch? If it looks OK, I'll commit to trunk and open CMR.
>>
>> ndex: btl_openib_component.c
>> ===================================================================
>> --- btl_openib_component.c (revision 26420)
>> +++ btl_openib_component.c (working copy)
>> @@ -810,8 +810,10 @@
>> we have to look up the values corresponding to
>> port->active_speed and port->active_width. These
>> are enums corresponding to the IB spec. Overall
>> - forumula is 80% of the reported speed (to get the
>> - true link speed) times the number of links. */
>> + forumula to get the true link speed is 8/10 or
>> + 64/66 of the reported speed (depends on the coding
>> + that is being used for the particular speed) times
>> + the number of links. */
>> switch (ib_port_attr->active_speed) {
>> case 1:
>> /* 2.5Gbps * 0.8, in megabits */
>> @@ -826,9 +828,13 @@
>> openib_btl->super.btl_bandwidth = 8000;
>> break;
>> case 8:
>> - /* 40.0Gbps * 0.8, in megabits */
>> - openib_btl->super.btl_bandwidth = 32000;
>> + /* FDR10: 10.0Gbps * 64/66, in megabits */
>> + openib_btl->super.btl_bandwidth = 9700;
>> break;
>> + case 16:
>> + /* FDR: 14.0Gbps * 64/66, in megabits */
>> + openib_btl->super.btl_bandwidth = 13600;
>> + break;
>> default:
>> /* Who knows? Declare this port unreachable (do
>> *not* return ERR_VALUE_OF_OUT_OF_BOUNDS; that
>
>