Open MPI logo

Open MPI Development Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Development mailing list

Subject: Re: [OMPI devel] [PATCH] fix mx btl_bandwidth
From: George Bosilca (bosilca_at_[hidden])
Date: 2010-09-03 11:33:42


On Sep 3, 2010, at 09:50 , Brice Goglin wrote:

> Le 03/09/2010 15:38, George Bosilca a écrit :
>> Jeff,
>>
>> I think you will have to revert this patch as the btl_bandwidth __IS__ supposed to be in Mbs and not MBs. We usually talk about networks in Mbs (there is a pattern in Ethernet 1G/10G, Myricom 10G). In addition the original design of the multi-rail was based on this assumption, and the multi-rail handling code deal with these values (at that level I don't think it really matters, but at least it needs consistent values from all BTLs).
>>
>> However, going over the existing BTLs I can see that some BTLs do not correctly set this value:
>>
>> BTL Bandwidth Auto-detect Status
>> Elan 2000 NO Correct
>>
>
> 2000 looks strange to me. Last time I played with Elan4, bandwidth was
> 900MB/s or so.

Lucky you ;) The 2000 was the bandwidth of the last Elan device we had.

>
>> GM 250 NO Doubtful
>> MX 2000/10000 YES (Mbs) Correct (before the patch)
>> OFUD 800 NO Doubtful
>> OpenIB 2000/4000/8000 YES (Mbs) Correct (multiplied by the active_width)
>>
>
> I found the problem when using both MX and OpenIB at the same time, so
> they can't be both wrong or both correct. IB was reporting 800, not
> 2000/4000/8000. Maybe because auto-detect didn't work and the default is
> wrong:
> btl_openib_mca.c:527: mca_btl_openib_module.super.btl_bandwidth = 800;

It appears that Open IB only auto-detect the bandwidth if the value is explicitly set to zero via the mca parameters. As a last resort: as for the other devices you can set it manually. Use something like btl_openib_bandwidth_%dev_name% to set the bandwidth per device.

  george.

>
> Brice
>
> _______________________________________________
> devel mailing list
> devel_at_[hidden]
> http://www.open-mpi.org/mailman/listinfo.cgi/devel