Open MPI logo

Open MPI Development Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Development mailing list

Subject: Re: [OMPI devel] openmpi-1.7.2 fails to use the RDMACM CPC
From: Jeff Squyres (jsquyres) (jsquyres_at_[hidden])
Date: 2013-08-19 21:18:26


Thanks for finding r27212. It was about a year ago, and had clearly fallen out of my cache (I have very little to do with the openib BTL these days).

Your solution isn't correct, because HAVE_IBV_LINK_LAYER_ETHERNET is defined (nor not) via this m4 macro in config/ompi_check_openfabrics.m4:

           AC_CHECK_DECLS([IBV_LINK_LAYER_ETHERNET],
                          [$1_have_rdmaoe=1], [],
                          [#include <infiniband/verbs.h>])

This m4 macro will #define HAVE_IBV_LINK_LAYER_ETHERNET if it exists, or #undef that name if it doesn't.

Do you not see the check for IBV_LINK_LAYER_ETHERNET in your configure stdout?

The code in the oob CPC in question is:

-----
    /* If we have the transport_type member, check to ensure we're on
       IB (this CPC will not work with iWarp). If we do not have the
       transport_type member, then we must be < OFED v1.2, and
       therefore we must be IB. */
#if defined(HAVE_STRUCT_IBV_DEVICE_TRANSPORT_TYPE) && defined(HAVE_IBV_LINK_LAYER_ETHERNET)
    if (BTL_OPENIB_CONNECT_BASE_CHECK_IF_NOT_IB(btl)) {
        opal_output_verbose(5, ompi_btl_base_framework.framework_output,
                            "openib BTL: oob CPC only supported on InfiniBand; skipped on %s:%d",
                            ibv_get_device_name(btl->device->ib_dev),
                            btl->port_num);
        return OMPI_ERR_NOT_SUPPORTED;
    }
#endif

----
So are you saying you have a libibverbs that does not have IBV_LINK_LAYER_ETHERNET, but it *does* support iWARP?
If so, as the comment clearly states, that would violate the assumption of that logic...  But I'm not sure how that could happen.
On Aug 19, 2013, at 5:38 PM, Steve Wise <swise_at_[hidden]> wrote:
> 
> 
>> -----Original Message-----
>> From: Steve Wise [mailto:swise_at_[hidden]]
>> Sent: Monday, August 19, 2013 4:02 PM
>> To: 'Open MPI Developers'; 'Jeff Squyres (jsquyres)'
>> Cc: 'Indranil Choudhury'
>> Subject: RE: [OMPI devel] openmpi-1.7.2 fails to use the RDMACM CPC
>> 
>> I guess HAVE_IBV_LINK_LAYER_ETHERNET is guarding against a libibverbs that doesn't have
>> IBV_LINK_LAYER_ETHERNET defined.  So the proper fix, I think, is to enhance configure to check
> this and
>> #define HAVE_IBV_LINK_LAYER_ETHERNET  if it exists.  Or have it check existence of a link_layer
> field in
>> the ibv_port_attr structure.
>> 
>> 
> 
> Maybe something like this?
> 
> Index: ompi_check_openfabrics.m4
> ===================================================================
> --- ompi_check_openfabrics.m4   (revision 29048)
> +++ ompi_check_openfabrics.m4   (working copy)
> @@ -198,7 +198,7 @@
>                           [#include <infiniband/verbs.h>])
> 
>            AC_MSG_CHECKING([if RDMAoE support is enabled])
> -           AC_DEFINE_UNQUOTED([OMPI_HAVE_RDMAOE], [$$1_have_rdmaoe], [Enable RDMAoE support])
> +           AC_DEFINE_UNQUOTED([HAVE_IBV_LINK_LAYER_ETHERNET], [$$1_have_rdmaoe], [Enable RDMAoE
> support])
>            if test "1" = "$$1_have_rdmaoe"; then
>                 AC_MSG_RESULT([yes])
>            else
> 
-- 
Jeff Squyres
jsquyres_at_[hidden]
For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/