Open MPI logo

Open MPI Development Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Development mailing list

Subject: Re: [OMPI devel] RFC: Linuxes shipping libibverbs
From: Brian W. Barrett (brbarret_at_[hidden])
Date: 2008-05-22 11:04:07


On Thu, 22 May 2008, Terry Dontje wrote:

>> The major difference here is that libmyriexpress is not being included
>> in mainline Linux distributions. Specifically: if you can find/use
>> libmyriexpress, it's likely because you have that hardware. The same
>> *used* to be true for libibverbs, but is no longer true because Linux
>> distros are now shipping (e.g., the Debian distribution pulls in
>> libibverbs when you install Open MPI).
>>
> Ok, but there are distributions that do include the myrinet BTL/MTL (ie
> CT). Though I agree for the most part in the case of myrinet if you
> have libmyriexpress you probably will probably have an operable
> interface. I guess I am curious how many other BTLs a distribution
> might end up delivering that could run into this reporting issue. I
> guess my point is could this be worth something more general instead of
> a one off for IB?
>
> From my point of view the btl_warn_unused_components coupled with "-mca
> btl ^mlfbtl" works for me. However the fact that the IB
> vendors/community (ie CISCO) is solving this for their favorite
> interface makes me pause for a moment.

There's actually a second (in my mind more important) reason why this is
IB only, as I shared similar concerns (hence yesterday's e-mail barage).
InfiniBand has a two stage initialization -- you get the list of HCAs,
then you initialize the HCA you want. So it's possible to determine that
there's no HCAs in the system vs. the system couldn't initialize the HCA
properly (as that would happen in step 2, according to Jeff).

With MX, it's one initialization call (mx_init), and it's not clear from
the errors it can return that you can differentiate between the two cases.
I haven't tried it, but it's possible that mx_init would succeed in the no
nic case, but then have a NIC count of 0.

Anyway, the short answer is that (in my opinion) we should have a btl base
param similar to warn_unused for whether to warn when no NICs/HCAs are
found, hopefully with a nice error function similar to today's no_nics
(which probably needs to be renamed in that case). That way, if BTL
authors other than OpenIB want to do some extra work and return better
error messages, they can.

>>> FWIW, our distribution actually turns off
>>> btl_base_want_component_unused
>>> because it seemed
>>> the majority of our cases would be that users would false positive
>>> sights of the message.
>>
>> Is the UDAPL library shipped in Solaris by default? If so, then
>> you're likely in exactly the same kind of situation that I'm
>> describing. The same will be true if Solaris ends up shipping
>> libibverbs by default.
>>
> Yes the UDAPL library is shipped in Solaris by default. Which is why we
> turn off
> btl_warn_unused_components. Yes, and I suspect once Solaris starts
> delivering libibverbs
> we (Sun) will need to figure out how to handle having both the udapl and
> openib btls being
> available.

There is some evil configure hackery that could be done to make this work
in a more general way (don't you love it when I say that).
Autogen/configure makes no guarantees about the order in which the
configure.m4 macros for components in the same framework are run, other
than all components of priority X are run before those of priority Y, iff
X > Y. So you could set the priority of all the components except udapl
to (say) 10 and udapl's to 0. Then have the udapl configure only build if
1) it was specifically requested or 2) ompi_check_openib_happy = no. No
more Linux-specific stuff, works when Solaris gets OFED, and works on old
Solaris that has uDAPL but not OFED.

As a matter of fact, it's so trivial to do that I'd recommend doing it for
1.3. Really, you could do it minimally by only changing OpenIB's
configure.params to set its priority to 10, uDAPL's configure.params to
set its priority to 0, and uDAPL's configure.m4 to remove the Linux stuff
and look for ompi_check_openib_happy.

Brian