Open MPI logo

Open MPI Development Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Development mailing list

Subject: Re: [OMPI devel] FDR: bad patch? (was: 1.6rc3 is out)
From: Yevgeny Kliteynik (kliteyn_at_[hidden])
Date: 2012-05-10 08:57:33


On 10-May-12 3:42 PM, Yevgeny Kliteynik wrote:
>
> Here are the ActiveSpeed values:
>
> 1: SDR - 2.5 Gb/s * (8/10 coding)
> 2: DDR - 5 Gb/s * (8/10 coding)
> 4: QDR - 10 Gb/s * (8/10 coding)
> 8: FDR10 - 10 Gb/s * (64/66 coding)
> 16: FDR - 14 Gbps * (64/66 coding)
> 32: EDR - 25 Gbs * (64/66 coding)
>
> EDR is not out yet, so we can ignore it for the sake of this patch.
> It is also true that *most* of IB cables are 4x.
> There are 12x clusters out there (like TACC's Ranger), and sometimes
> there are 1x links (mostly because the cable is not perfect or too long).
>
> So cases of 4 and 8 are not the same for us, because MPI needs actual
> *data* rate instead of *bit* rate, which is different in these cases.
>
> I'll send a patch for review in a minute...
>

How about this patch? If it looks OK, I'll commit to trunk and open CMR.

ndex: btl_openib_component.c
===================================================================
--- btl_openib_component.c (revision 26420)
+++ btl_openib_component.c (working copy)
@@ -810,8 +810,10 @@
                    we have to look up the values corresponding to
                    port->active_speed and port->active_width. These
                    are enums corresponding to the IB spec. Overall
- forumula is 80% of the reported speed (to get the
- true link speed) times the number of links. */
+ forumula to get the true link speed is 8/10 or
+ 64/66 of the reported speed (depends on the coding
+ that is being used for the particular speed) times
+ the number of links. */
                 switch (ib_port_attr->active_speed) {
                 case 1:
                     /* 2.5Gbps * 0.8, in megabits */
@@ -826,9 +828,13 @@
                     openib_btl->super.btl_bandwidth = 8000;
                     break;
                 case 8:
- /* 40.0Gbps * 0.8, in megabits */
- openib_btl->super.btl_bandwidth = 32000;
+ /* FDR10: 10.0Gbps * 64/66, in megabits */
+ openib_btl->super.btl_bandwidth = 9700;
                     break;
+ case 16:
+ /* FDR: 14.0Gbps * 64/66, in megabits */
+ openib_btl->super.btl_bandwidth = 13600;
+ break;
                 default:
                     /* Who knows? Declare this port unreachable (do
                        *not* return ERR_VALUE_OF_OUT_OF_BOUNDS; that