Open MPI logo

Open MPI Development Mailing List Archives

  |   Home   |   Support   |   FAQ   |  

This web mail archive is frozen.

This page is part of a frozen web archive of this mailing list.

You can still navigate around this archive, but know that no new mails have been added to it since July of 2016.

Click here to be taken to the new web archives of this list; it includes all the mails that are in this frozen archive plus all new mails that have been sent to the list since it was migrated to the new archives.

Subject: Re: [OMPI devel] [OMPI svn-full] svn:open-mpi r24830
From: Yevgeny Kliteynik (kliteyn_at_[hidden])
Date: 2011-07-13 19:20:22


Hi Jeff,

Just checked in r24900.

On 09-Jul-11 2:24 PM, Jeff Squyres wrote:
> I'm still getting warnings about OMPI_ENABLE_DYNAMIC_SL not being defined, even though I see this in my configure output:
>
> checking if can use dynamic SL support... yes
>
> That's why I wanted that macro *always* defined by the m4 (to either 0 or 1) -- not just defining it or not. This is one of the OMPI coding guidelines: always define logical preprocessor macros to 0 or 1, not define-them-or-not. If you always define them, then you can get preprocessor warnings if you misspell a macro name. E.g.:
>
> #if MISSPELLED_MACRO_NAME
>
> will return a warning because the macro doesn't exist. But if you just define-the-macro-or-not and use
>
> #ifdef MISSPELLED_MACRO_NAME
>
> then you won't get a warning and it may be a difficult-to-find bug that you misspelled the macro name in the code.

OK, got it. Fixed.

> Also, note that the 4-argument version of AC_ARG_ENABLE isn't really necessary:
>
> AC_ARG_ENABLE([openib-dynamic-sl],
> [AC_HELP_STRING([--enable-openib-dynamic-sl],
> [Enable openib BTL to query Subnet Manager for IB SL (default: enabled)])],
> [enable_openib_dynamic_sl="$enableval"],
> [enable_openib_dynamic_sl="not_provided"])
>
> You can shorten it to:
>
> AC_ARG_ENABLE([openib-dynamic-sl],
> [AC_HELP_STRING([--enable-openib-dynamic-sl],
> [Enable openib BTL to query Subnet Manager for IB SL (default: enabled)])])
>
> because $enable_openib_dynamic_sl will automatically be set to "yes" (when --enable-openib-dynamic-sl is used), "no" (when --disable-openib-dynamic-sl is used), or "" (when neither is used). So there's no need to manually set this variable in args 3 and 4 -- it's set for you automatically.
>
> I know we have some of the older 4-arg forms in ompi_check_openib.m4, but they're just old and haven't been updated. You probably don't want to introduce *new* 4-arg usage.

Guess I was copying from one of those old usages.
Fixed.
 
> Finally, are you sure that infiniband/complib/cl_types_osd.h exists on all platforms? (e.g., Solaris) I know you said you don't have any Solaris machines to test with, but you should ping Oracle directly for some testing -- Terry might not be paying attention to this specific thread...

I'll check it, but my guess would be that Solaris doesn't have it.
AFAIK Solaris doesn't use OpenFabrics OpenSM - it has a separate
subnet manager, so I can't assume that it has.
So right now the dynamic SL will probably not work on Solaris
(though it won't break the compilation).

>
> -----
>
> Here's the warnings I'm seeing -- did you remove the AC_DEFINE for OMPI_ENABLE_DYNAMIC_SL altogether by accident?

It appears that I did lose it...
I'm spoiled by git, hard to get used to svn again... :)

-- YK

> [4:13] svbu-mpi:~/svn/ompi5/ompi/mca/btl/openib % make
> CC btl_openib.lo
> In file included from btl_openib_ini.h:16:0,
> from btl_openib.c:47:
> btl_openib.h:219:6: warning: "OMPI_ENABLE_DYNAMIC_SL" is not defined
> CC btl_openib_component.lo
> In file included from btl_openib_component.c:80:0:
> btl_openib.h:219:6: warning: "OMPI_ENABLE_DYNAMIC_SL" is not defined
> CC btl_openib_endpoint.lo
> In file included from btl_openib_endpoint.h:32:0,
> from btl_openib_endpoint.c:46:
> btl_openib.h:219:6: warning: "OMPI_ENABLE_DYNAMIC_SL" is not defined
> CC btl_openib_frag.lo
> In file included from btl_openib_frag.c:22:0:
> btl_openib.h:219:6: warning: "OMPI_ENABLE_DYNAMIC_SL" is not defined
> CC btl_openib_proc.lo
> In file included from btl_openib_proc.c:27:0:
> btl_openib.h:219:6: warning: "OMPI_ENABLE_DYNAMIC_SL" is not defined
> CC btl_openib_lex.lo
> btl_openib_lex.c: In function 'yy_get_next_buffer':
> btl_openib_lex.c:1229:3: warning: comparison between signed and unsigned integer expressions
> btl_openib_lex.l: At top level:
> btl_openib_lex.c:1323:17: warning: 'yyunput' defined but not used
> btl_openib_lex.c:1364:16: warning: 'input' defined but not used
> CC btl_openib_mca.lo
> In file included from btl_openib_mca.c:33:0:
> btl_openib.h:219:6: warning: "OMPI_ENABLE_DYNAMIC_SL" is not defined
> btl_openib_mca.c: In function 'btl_openib_register_mca_params':
> btl_openib_mca.c:401:6: warning: "OMPI_ENABLE_DYNAMIC_SL" is not defined
> CC btl_openib_ini.lo
> In file included from btl_openib_ini.c:35:0:
> btl_openib.h:219:6: warning: "OMPI_ENABLE_DYNAMIC_SL" is not defined
> CC btl_openib_async.lo
> In file included from btl_openib_async.c:26:0:
> btl_openib.h:219:6: warning: "OMPI_ENABLE_DYNAMIC_SL" is not defined
> CC btl_openib_xrc.lo
> In file included from btl_openib_xrc.h:14:0,
> from btl_openib_xrc.c:23:
> btl_openib.h:219:6: warning: "OMPI_ENABLE_DYNAMIC_SL" is not defined
> CC btl_openib_fd.lo
> CC btl_openib_ip.lo
> In file included from btl_openib_endpoint.h:32:0,
> from btl_openib_ip.c:30:
> btl_openib.h:219:6: warning: "OMPI_ENABLE_DYNAMIC_SL" is not defined
> CC connect/btl_openib_connect_base.lo
> In file included from connect/btl_openib_connect_base.c:13:0:
> ./btl_openib.h:219:6: warning: "OMPI_ENABLE_DYNAMIC_SL" is not defined
> CC connect/btl_openib_connect_oob.lo
> In file included from connect/btl_openib_connect_oob.c:41:0:
> ./btl_openib.h:219:6: warning: "OMPI_ENABLE_DYNAMIC_SL" is not defined
> connect/btl_openib_connect_oob.c:47:6: warning: "OMPI_ENABLE_DYNAMIC_SL" is not defined
> connect/btl_openib_connect_oob.c:65:6: warning: "OMPI_ENABLE_DYNAMIC_SL" is not defined
> connect/btl_openib_connect_oob.c:115:6: warning: "OMPI_ENABLE_DYNAMIC_SL" is not defined
> connect/btl_openib_connect_oob.c:271:6: warning: "OMPI_ENABLE_DYNAMIC_SL" is not defined
> connect/btl_openib_connect_oob.c: In function 'oob_component_finalize':
> connect/btl_openib_connect_oob.c:307:7: warning: "OMPI_ENABLE_DYNAMIC_SL" is not defined
> connect/btl_openib_connect_oob.c: In function 'qp_connect_all':
> connect/btl_openib_connect_oob.c:396:6: warning: "OMPI_ENABLE_DYNAMIC_SL" is not defined
> connect/btl_openib_connect_oob.c: At top level:
> connect/btl_openib_connect_oob.c:1011:6: warning: "OMPI_ENABLE_DYNAMIC_SL" is not defined
> CC connect/btl_openib_connect_empty.lo
> In file included from connect/btl_openib_connect_empty.c:13:0:
> ./btl_openib.h:219:6: warning: "OMPI_ENABLE_DYNAMIC_SL" is not defined
> CC connect/btl_openib_connect_xoob.lo
> In file included from connect/btl_openib_connect_xoob.c:30:0:
> ./btl_openib.h:219:6: warning: "OMPI_ENABLE_DYNAMIC_SL" is not defined
> CC connect/btl_openib_connect_rdmacm.lo
> In file included from ./btl_openib_proc.h:26:0,
> from connect/btl_openib_connect_rdmacm.c:53:
> ./btl_openib.h:219:6: warning: "OMPI_ENABLE_DYNAMIC_SL" is not defined
> CCLD libmca_btl_openib.la
> [4:13] svbu-mpi:~/svn/ompi5/ompi/mca/btl/openib % cd
>
>
>
> On Jul 3, 2011, at 11:07 AM, Yevgeny Kliteynik wrote:
>
>> Hi Jeff,
>>
>> On 02-Jul-11 11:52 PM, Jeff Squyres (jsquyres) wrote:
>>> Were all the issueswith this code fixed? There were m4 issues and solaris issues, IIRC.
>>
>> I took all the fixes I could find based on the trac:
>> "Be sure also to look at r24196; Josh committed a
>> bunch of warning fixes for you after r24915"
>>
>> I also removed all the libibmad dependencies and unneded macros,
>> so I hope that this is OK. However, I don't have any Solaris machine
>> to try this to make sure that there are no issues.
>>
>> The only complaint w.r.t. Solaris that I could find was Terry's
>> mail from last week, but it turned out to be a different problem.
>>
>> Are there any other problems that I'm not aware of?
>>
>> -- YK
>>
>>
>>> Sent from my phone. No type good.
>>>
>>> On Jun 28, 2011, at 9:28 AM, "kliteyn_at_[hidden]"<kliteyn_at_[hidden]> wrote:
>>>
>>>> Author: kliteyn
>>>> Date: 2011-06-28 10:28:29 EDT (Tue, 28 Jun 2011)
>>>> New Revision: 24830
>>>> URL: https://svn.open-mpi.org/trac/ompi/changeset/24830
>>>>
>>>> Log:
>>>> Supporting dynamic SL (#2674)
>>>>
>>>> - Added enable/disable configuration parameter for dynamic SL
>>>> - All the dynamic SL code is conditionalized
>>>> - Removed libibmad dependency
>>>> - Using only one include - ib_types.h (part of opensm-devel package)
>>>> - Removed all the macro and data types definitions, using the
>>>> existing definitions from ib_types.h instead
>>>> - general cleaning here and there
>>>>
>>>> The async mode is not implemented yet - stay tuned...
>>>>
>>>>
>>>> Text files modified:
>>>> trunk/ompi/config/ompi_check_openib.m4 | 38 ++++
>>>> trunk/ompi/mca/btl/openib/btl_openib.h | 5
>>>> trunk/ompi/mca/btl/openib/btl_openib_mca.c | 10
>>>> trunk/ompi/mca/btl/openib/connect/btl_openib_connect_oob.c | 309 +++++++++++++++++----------------------
>>>> 4 files changed, 182 insertions(+), 180 deletions(-)
>>>>
>>>> Modified: trunk/ompi/config/ompi_check_openib.m4
>>>> ==============================================================================
>>>> --- trunk/ompi/config/ompi_check_openib.m4 (original)
>>>> +++ trunk/ompi/config/ompi_check_openib.m4 2011-06-28 10:28:29 EDT (Tue, 28 Jun 2011)
>>>> @@ -155,11 +155,21 @@
>>>> [$ompi_cv_func_ibv_create_cq_args],
>>>> [Number of arguments to ibv_create_cq])])])
>>>>
>>>> + #
>>>> + # OpenIB dynamic SL
>>>> + #
>>>> + AC_ARG_ENABLE([openib-dynamic-sl],
>>>> + [AC_HELP_STRING([--enable-openib-dynamic-sl],
>>>> + [Enable openib BTL to query Subnet Manager for IB SL (default: enabled)])],
>>>> + [enable_openib_dynamic_sl="$enableval"],
>>>> + [enable_openib_dynamic_sl="yes"])
>>>> +
>>>> # Set these up so that we can do an AC_DEFINE below
>>>> # (unconditionally)
>>>> $1_have_xrc=0
>>>> $1_have_rdmacm=0
>>>> $1_have_ibcm=0
>>>> + $1_have_dynamic_sl=0
>>>>
>>>> # If we have the openib stuff available, find out what we've got
>>>> AS_IF([test "$ompi_check_openib_happy" = "yes"],
>>>> @@ -176,6 +186,19 @@
>>>> AC_CHECK_FUNCS([ibv_create_xrc_rcv_qp], [$1_have_xrc=1])
>>>> fi
>>>>
>>>> + if test "$enable_openib_dynamic_sl" = "yes"; then
>>>> + # We need ib_types.h file, which is installed with opensm-devel
>>>> + # package. However, ib_types.h has a bad include directive,
>>>> + # which will cause AC_CHECK_HEADER to fail.
>>>> + # So instead, we will look for another file that is also
>>>> + # installed as part of opensm-devel package and included in
>>>> + # ib_types.h, but it doesn't include any other IB-related files.
>>>> + AC_CHECK_HEADER([infiniband/complib/cl_types_osd.h],
>>>> + [$1_have_dynamic_sl=1],
>>>> + [AC_MSG_ERROR([opensm-devel package not found - please install it or disable dynamic SL support with \"--disable-openib-dynamic-sl\"])],
>>>> + [])
>>>> + fi
>>>> +
>>>> # Do we have a recent enough RDMA CM? Need to have the
>>>> # rdma_get_peer_addr (inline) function (originally appeared
>>>> # in OFED v1.3).
>>>> @@ -244,6 +267,15 @@
>>>> else
>>>> AC_MSG_RESULT([no])
>>>> fi
>>>> +
>>>> + AC_MSG_CHECKING([if dynamic SL is enabled])
>>>> + AC_DEFINE_UNQUOTED([OMPI_ENABLE_DYNAMIC_SL], [$$1_have_dynamic_sl],
>>>> + [Enable features required for dynamic SL support])
>>>> + if test "1" = "$$1_have_dynamic_sl"; then
>>>> + AC_MSG_RESULT([yes])
>>>> + else
>>>> + AC_MSG_RESULT([no])
>>>> + fi
>>>>
>>>> AC_MSG_CHECKING([if OpenFabrics RDMACM support is enabled])
>>>> AC_DEFINE_UNQUOTED([OMPI_HAVE_RDMACM], [$$1_have_rdmacm],
>>>> @@ -267,7 +299,11 @@
>>>> AC_MSG_RESULT([no])
>>>> fi
>>>>
>>>> - CPPFLAGS="$ompi_check_openib_$1_save_CPPFLAGS"
>>>> + AS_IF([test -z "$ompi_check_openib_dir"],
>>>> + [openib_include_dir="/usr/include"],
>>>> + [openib_include_dir="$ompi_check_openib_dir/include"])
>>>> +
>>>> + CPPFLAGS="$ompi_check_openib_$1_save_CPPFLAGS -I$openib_include_dir/infiniband"
>>>> LDFLAGS="$ompi_check_openib_$1_save_LDFLAGS"
>>>> LIBS="$ompi_check_openib_$1_save_LIBS"
>>>>
>>>>
>>>> Modified: trunk/ompi/mca/btl/openib/btl_openib.h
>>>> ==============================================================================
>>>> --- trunk/ompi/mca/btl/openib/btl_openib.h (original)
>>>> +++ trunk/ompi/mca/btl/openib/btl_openib.h 2011-06-28 10:28:29 EDT (Tue, 28 Jun 2011)
>>>> @@ -52,6 +52,7 @@
>>>> BEGIN_C_DECLS
>>>>
>>>> #define HAVE_XRC (1 == OMPI_HAVE_CONNECTX_XRC)
>>>> +#define ENABLE_DYNAMIC_SL (1 == OMPI_ENABLE_DYNAMIC_SL)
>>>>
>>>> #define MCA_BTL_IB_LEAVE_PINNED 1
>>>> #define IB_DEFAULT_GID_PREFIX 0xfe80000000000000ll
>>>> @@ -215,7 +216,9 @@
>>>> uint32_t ib_rnr_retry;
>>>> uint32_t ib_max_rdma_dst_ops;
>>>> uint32_t ib_service_level;
>>>> - uint32_t ib_path_rec_service_level;
>>>> +#if (ENABLE_DYNAMIC_SL)
>>>> + uint32_t ib_path_record_service_level;
>>>> +#endif
>>>> int32_t use_eager_rdma;
>>>> int32_t eager_rdma_threshold; /**< After this number of msg, use RDMA for short messages, always */
>>>> int32_t eager_rdma_num;
>>>>
>>>> Modified: trunk/ompi/mca/btl/openib/btl_openib_mca.c
>>>> ==============================================================================
>>>> --- trunk/ompi/mca/btl/openib/btl_openib_mca.c (original)
>>>> +++ trunk/ompi/mca/btl/openib/btl_openib_mca.c 2011-06-28 10:28:29 EDT (Tue, 28 Jun 2011)
>>>> @@ -398,10 +398,14 @@
>>>> }
>>>> mca_btl_openib_component.ib_service_level = (uint32_t) ival;
>>>>
>>>> - CHECK(reg_int("ib_path_rec_service_level", NULL, "Enable getting InfiniBand service level from PathRecord "
>>>> - "(must be>= 0, 0 = disabled, positive = try to get the service level from PathRecord)",
>>>> +#if (ENABLE_DYNAMIC_SL)
>>>> + CHECK(reg_int("ib_path_record_service_level", NULL,
>>>> + "Enable getting InfiniBand service level from PathRecord "
>>>> + "(must be>= 0, 0 = disabled, positive = try to get the "
>>>> + "service level from PathRecord)",
>>>> 0,&ival, REGINT_GE_ZERO));
>>>> - mca_btl_openib_component.ib_path_rec_service_level = (uint32_t) ival;
>>>> + mca_btl_openib_component.ib_path_record_service_level = (uint32_t) ival;
>>>> +#endif
>>>>
>>>> CHECK(reg_int("use_eager_rdma", NULL, "Use RDMA for eager messages "
>>>> "(-1 = use device default, 0 = do not use eager RDMA, "
>>>>
>>>> Modified: trunk/ompi/mca/btl/openib/connect/btl_openib_connect_oob.c
>>>> ==============================================================================
>>>> --- trunk/ompi/mca/btl/openib/connect/btl_openib_connect_oob.c (original)
>>>> +++ trunk/ompi/mca/btl/openib/connect/btl_openib_connect_oob.c 2011-06-28 10:28:29 EDT (Tue, 28 Jun 2011)
>>>> @@ -44,6 +44,10 @@
>>>> #include "connect/connect.h"
>>>> #include "orte/util/show_help.h"
>>>>
>>>> +#if (ENABLE_DYNAMIC_SL)
>>>> +#include<infiniband/iba/ib_types.h>
>>>> +#endif
>>>> +
>>>> #ifdef HAVE_UNISTD_H
>>>> #include<unistd.h>
>>>> #endif
>>>> @@ -54,109 +58,17 @@
>>>> ENDPOINT_CONNECT_ACK
>>>> } connect_message_type_t;
>>>>
>>>> -#ifndef __WINDOWS__
>>>> -#define PACK_SUFFIX __attribute__((packed))
>>>> -#else
>>>> -#define PACK_SUFFIX
>>>> -#endif
>>>> -
>>>> -#define SL_NOT_PRESENT 0x7F
>>>> +#define SL_NOT_PRESENT 0xFF
>>>> #define MAX_GET_SL_REC_RETRIES 20
>>>> #define GET_SL_REC_RETRIES_TIMEOUT_MS 2000000
>>>>
>>>> -#define IB_SA_QPN 1
>>>> -#define IB_GLOBAL_QKEY 0x80010000UL
>>>> -#define IB_MGMT_BASE_VERSION 1
>>>> -#define IB_MGMT_CLASS_SUBN_ADM 0x03
>>>> -#define IB_MGMT_METHOD_GET 0x01
>>>> -#define IB_SA_TID_GET_PATH_REC_0 0xCA000000UL
>>>> -#define IB_SA_TID_GET_PATH_REC_1 0xBEEF0000UL
>>>> -#define IB_PATH_REC_SL_MASK 0x000F
>>>> -#define IB_SA_ATTR_PATH_REC 0x35
>>>> -#define IB_SA_PATH_REC_DLID (1<<4)
>>>> -#define IB_SA_PATH_REC_SLID (1<<5)
>>>> -
>>>> -
>>>> -#ifdef __WINDOWS__
>>>> - #pragma pack(push)
>>>> - #pragma pack(1)
>>>> -#endif
>>>> -
>>>> -struct ib_mad_hdr {
>>>> - uint8_t base_version;
>>>> - uint8_t mgmt_class;
>>>> - uint8_t class_version;
>>>> - uint8_t method;
>>>> - uint16_t status;
>>>> - uint16_t class_spec;
>>>> - uint32_t tid[2];
>>>> - uint16_t attr_id;
>>>> - uint16_t resv;
>>>> - uint32_t attr_mod;
>>>> -} PACK_SUFFIX;
>>>> -
>>>> -struct ib_rmpp_hdr {
>>>> - uint32_t raw[3];
>>>> -} PACK_SUFFIX;
>>>> -
>>>> -struct ib_sa_hdr {
>>>> - uint32_t sm_key[2];
>>>> - uint16_t reserved;
>>>> - uint16_t attrib_offset;
>>>> - uint32_t comp_mask[2];
>>>> -} PACK_SUFFIX;
>>>> -
>>>> -typedef union _ib_gid {
>>>> - uint8_t raw[16];
>>>> - struct _ib_gid_unicast {
>>>> - uint64_t prefix;
>>>> - uint64_t interface_id;
>>>> - } PACK_SUFFIX unicast;
>>>> - struct _ib_gid_multicast {
>>>> - uint8_t header[2];
>>>> - uint8_t raw_group_id[14];
>>>> - } PACK_SUFFIX multicast;
>>>> -} PACK_SUFFIX ib_gid_t;
>>>> -
>>>> -struct ib_path_record {
>>>> - uint64_t service_id;
>>>> - ib_gid_t dgit;
>>>> - ib_gid_t sgit;
>>>> - uint16_t dlid;
>>>> - uint16_t slid;
>>>> - uint32_t hop_flow_raw;
>>>> - uint8_t tclass;
>>>> - uint8_t num_path;
>>>> - uint16_t pkey;
>>>> - uint8_t reserved1;
>>>> - uint8_t qos_class_sl;
>>>> - uint8_t mtu;
>>>> - uint8_t rate;
>>>> - uint32_t preference__packet_lifetime__packet_lifetime_selector;
>>>> - uint32_t reserved2[35];
>>>> -} PACK_SUFFIX;
>>>> -
>>>> -union ib_sa_data {
>>>> - struct ib_path_record path_record;
>>>> -} PACK_SUFFIX;
>>>> -
>>>> -struct ib_mad_sa {
>>>> - struct ib_mad_hdr mad_hdr;
>>>> - struct ib_rmpp_hdr rmpp_hdr;
>>>> - struct ib_sa_hdr sa_hdr;
>>>> - union ib_sa_data sa_data;
>>>> -} PACK_SUFFIX;
>>>> -
>>>> -#ifdef __WINDOWS__
>>>> - #pragma pack(pop)
>>>> -#endif
>>>> -
>>>> +#if (ENABLE_DYNAMIC_SL)
>>>> static struct mca_btl_openib_sa_qp_cache {
>>>> /* There will be a MR with the one send and receive buffer together */
>>>> /* The send buffer is first, the receive buffer is second */
>>>> /* The receive buffer in a UD queue pair needs room for the 40 byte GRH */
>>>> /* The buffers are first in the structure for page alignment */
>>>> - char send_recv_buffer[sizeof(struct ib_mad_sa) * 2 + 40];
>>>> + char send_recv_buffer[MAD_BLOCK_SIZE * 2 + 40];
>>>> struct mca_btl_openib_sa_qp_cache *next;
>>>> struct ibv_context *context;
>>>> char *device_name;
>>>> @@ -168,8 +80,9 @@
>>>> struct ibv_pd *pd;
>>>> struct ibv_recv_wr rwr;
>>>> struct ibv_sge rsge;
>>>> - char sl_values[65536];
>>>> + uint8_t sl_values[65536]; /* 64K */
>>>> } *sa_qp_cache = 0;
>>>> +#endif
>>>>
>>>> static int oob_priority = 50;
>>>> static bool rml_recv_posted = false;
>>>> @@ -198,27 +111,31 @@
>>>> static void rml_recv_cb(int status, orte_process_name_t* process_name,
>>>> opal_buffer_t* buffer, orte_rml_tag_t tag,
>>>> void* cbdata);
>>>> +
>>>> +#if (ENABLE_DYNAMIC_SL)
>>>> static int init_ud_qp(struct ibv_context *context_arg,
>>>> struct mca_btl_openib_sa_qp_cache *cache);
>>>> static void init_sa_mad(struct mca_btl_openib_sa_qp_cache *cache,
>>>> - struct ib_mad_sa *sag,
>>>> - struct ibv_send_wr *swr,
>>>> - struct ibv_sge *ssge,
>>>> - uint16_t lid,
>>>> - uint16_t rem_lid);
>>>> + ib_sa_mad_t *sa_mad,
>>>> + struct ibv_send_wr *swr,
>>>> + struct ibv_sge *ssge,
>>>> + uint16_t lid,
>>>> + uint16_t rem_lid);
>>>> static int get_pathrecord_info(struct mca_btl_openib_sa_qp_cache *cache,
>>>> - struct ib_mad_sa *sag,
>>>> - struct ib_mad_sa *sar,
>>>> - struct ibv_send_wr *swr,
>>>> - uint16_t lid,
>>>> - uint16_t rem_lid);
>>>> -static int init_device(struct ibv_context *context_arg,
>>>> - struct mca_btl_openib_sa_qp_cache *cache,
>>>> - uint32_t port_num);
>>>> -static int get_pathrecord_sl(struct ibv_context *context_arg,
>>>> - uint32_t port_num,
>>>> + ib_sa_mad_t *sa_mad,
>>>> + ib_sa_mad_t *sar,
>>>> + struct ibv_send_wr *swr,
>>>> uint16_t lid,
>>>> uint16_t rem_lid);
>>>> +static int init_device(struct ibv_context *context_arg,
>>>> + struct mca_btl_openib_sa_qp_cache *cache,
>>>> + uint32_t port_num);
>>>> +static int get_pathrecord_sl(struct ibv_context *context_arg,
>>>> + uint32_t port_num,
>>>> + uint16_t lid,
>>>> + uint16_t rem_lid);
>>>> +static void free_sa_qp_cache(void);
>>>> +#endif
>>>>
>>>> /*
>>>> * The "component" struct -- the top-level function pointers for the
>>>> @@ -351,6 +268,33 @@
>>>> return OMPI_SUCCESS;
>>>> }
>>>>
>>>> +#if (ENABLE_DYNAMIC_SL)
>>>> +static void free_sa_qp_cache(void)
>>>> +{
>>>> + struct mca_btl_openib_sa_qp_cache *cache, *tmp;
>>>> +
>>>> + cache = sa_qp_cache;
>>>> + while (NULL != cache) {
>>>> + /* free cache data */
>>>> + if (cache->device_name)
>>>> + free(cache->device_name);
>>>> + if (NULL != cache->qp)
>>>> + ibv_destroy_qp(cache->qp);
>>>> + if (NULL != cache->ah)
>>>> + ibv_destroy_ah(cache->ah);
>>>> + if (NULL != cache->cq)
>>>> + ibv_destroy_cq(cache->cq);
>>>> + if (NULL != cache->mr)
>>>> + ibv_dereg_mr(cache->mr);
>>>> + if (NULL != cache->pd)
>>>> + ibv_dealloc_pd(cache->pd);
>>>> + tmp = cache->next;
>>>> + free(cache);
>>>> + cache = tmp;
>>>> + }
>>>> +}
>>>> +#endif
>>>> +
>>>> /*
>>>> * Component finalize function. Cleanup RML non-blocking receive.
>>>> */
>>>> @@ -360,7 +304,9 @@
>>>> orte_rml.recv_cancel(ORTE_NAME_WILDCARD, OMPI_RML_TAG_OPENIB);
>>>> rml_recv_posted = false;
>>>> }
>>>> -
>>>> + #if (ENABLE_DYNAMIC_SL)
>>>> + free_sa_qp_cache();
>>>> +#endif
>>>> return OMPI_SUCCESS;
>>>> }
>>>>
>>>> @@ -425,7 +371,7 @@
>>>> */
>>>> static int qp_connect_all(mca_btl_openib_endpoint_t *endpoint)
>>>> {
>>>> - int i, rc;
>>>> + int i;
>>>> mca_btl_openib_module_t* openib_btl =
>>>> (mca_btl_openib_module_t*)endpoint->endpoint_btl;
>>>>
>>>> @@ -446,18 +392,24 @@
>>>> attr.ah_attr.dlid = endpoint->rem_info.rem_lid;
>>>> attr.ah_attr.src_path_bits = openib_btl->src_path_bits;
>>>> attr.ah_attr.port_num = openib_btl->port_num;
>>>> - attr.ah_attr.sl = mca_btl_openib_component.ib_service_level;
>>>> - /* if user enable ib_path_rec_service_level - dynamically get the sl from PathRecord */
>>>> - if (mca_btl_openib_component.ib_path_rec_service_level> 0) {
>>>> - rc = get_pathrecord_sl(qp->context,
>>>> +
>>>> +#if (ENABLE_DYNAMIC_SL)
>>>> + /* if user enabled dynamic SL, get it from PathRecord */
>>>> + if (0 != mca_btl_openib_component.ib_path_record_service_level) {
>>>> + int rc = get_pathrecord_sl(qp->context,
>>>> attr.ah_attr.port_num,
>>>> openib_btl->lid,
>>>> attr.ah_attr.dlid);
>>>> if (OMPI_ERROR == rc) {
>>>> + free_sa_qp_cache();
>>>> return OMPI_ERROR;
>>>> }
>>>> attr.ah_attr.sl = rc;
>>>> }
>>>> +#else
>>>> + attr.ah_attr.sl = mca_btl_openib_component.ib_service_level;
>>>> +#endif
>>>> +
>>>> /* JMS to be filled in later dynamically */
>>>> attr.ah_attr.static_rate = 0;
>>>>
>>>> @@ -1056,6 +1008,7 @@
>>>> OPAL_THREAD_UNLOCK(&mca_btl_openib_component.ib_lock);
>>>> }
>>>>
>>>> +#if (ENABLE_DYNAMIC_SL)
>>>> static int init_ud_qp(struct ibv_context *context_arg,
>>>> struct mca_btl_openib_sa_qp_cache *cache)
>>>> {
>>>> @@ -1094,7 +1047,7 @@
>>>> memset(&mattr, 0, sizeof(mattr));
>>>> mattr.qp_state = IBV_QPS_INIT;
>>>> mattr.port_num = cache->port_num;
>>>> - mattr.qkey = IB_GLOBAL_QKEY;
>>>> + mattr.qkey = ntohl(IB_QP1_WELL_KNOWN_Q_KEY);
>>>> rc = ibv_modify_qp(cache->qp,&mattr,
>>>> IBV_QP_STATE |
>>>> IBV_QP_PKEY_INDEX |
>>>> @@ -1128,61 +1081,75 @@
>>>> return OMPI_SUCCESS;
>>>> }
>>>> static void init_sa_mad(struct mca_btl_openib_sa_qp_cache *cache,
>>>> - struct ib_mad_sa *sag,
>>>> - struct ibv_send_wr *swr,
>>>> - struct ibv_sge *ssge,
>>>> - uint16_t lid,
>>>> - uint16_t rem_lid)
>>>> + ib_sa_mad_t *sa_mad,
>>>> + struct ibv_send_wr *swr,
>>>> + struct ibv_sge *ssge,
>>>> + uint16_t lid,
>>>> + uint16_t rem_lid)
>>>> {
>>>> - memset(sag, 0, sizeof(*sag));
>>>> + ib_path_rec_t *path_record = (ib_path_rec_t*)sa_mad->data;
>>>> +
>>>> memset(swr, 0, sizeof(*swr));
>>>> memset(ssge, 0, sizeof(*ssge));
>>>>
>>>> - sag->mad_hdr.base_version = IB_MGMT_BASE_VERSION;
>>>> - sag->mad_hdr.mgmt_class = IB_MGMT_CLASS_SUBN_ADM;
>>>> - sag->mad_hdr.class_version = 2;
>>>> - sag->mad_hdr.method = IB_MGMT_METHOD_GET;
>>>> - sag->mad_hdr.attr_id = htons (IB_SA_ATTR_PATH_REC);
>>>> - sag->mad_hdr.tid[0] = IB_SA_TID_GET_PATH_REC_0 + cache->qp->qp_num;
>>>> - sag->mad_hdr.tid[1] = IB_SA_TID_GET_PATH_REC_1 + rem_lid;
>>>> - sag->sa_hdr.comp_mask[1] =
>>>> - htonl(IB_SA_PATH_REC_DLID | IB_SA_PATH_REC_SLID);
>>>> - sag->sa_data.path_record.dlid = htons(rem_lid);
>>>> - sag->sa_data.path_record.slid = htons(lid);
>>>> + /* Initialize the standard MAD header. */
>>>> + memset(sa_mad, 0, MAD_BLOCK_SIZE);
>>>> + ib_mad_init_new((ib_mad_t *)sa_mad, /* mad header pointer */
>>>> + IB_MCLASS_SUBN_ADM, /* management class */
>>>> + (uint8_t) 2, /* version */
>>>> + IB_MAD_METHOD_GET, /* method */
>>>> + hton64((uint64_t)lid<< 48 | /* transaction ID */
>>>> + (uint64_t)rem_lid<< 32 |
>>>> + (uint64_t)cache->qp->qp_num<< 8),
>>>> + IB_MAD_ATTR_PATH_RECORD, /* attribute ID */
>>>> + 0); /* attribute modifier */
>>>> +
>>>> + sa_mad->comp_mask = IB_PR_COMPMASK_DLID | IB_PR_COMPMASK_SLID;
>>>> + path_record->dlid = htons(rem_lid);
>>>> + path_record->slid = htons(lid);
>>>>
>>>> swr->sg_list = ssge;
>>>> swr->num_sge = 1;
>>>> swr->opcode = IBV_WR_SEND;
>>>> swr->wr.ud.ah = cache->ah;
>>>> - swr->wr.ud.remote_qpn = IB_SA_QPN;
>>>> - swr->wr.ud.remote_qkey = IB_GLOBAL_QKEY;
>>>> + swr->wr.ud.remote_qpn = ntohl(IB_QP1);
>>>> + swr->wr.ud.remote_qkey = ntohl(IB_QP1_WELL_KNOWN_Q_KEY);
>>>> swr->send_flags = IBV_SEND_SIGNALED | IBV_SEND_SOLICITED;
>>>>
>>>> - ssge->addr = (uint64_t)(void *)sag;
>>>> - ssge->length = sizeof(*sag);
>>>> + ssge->addr = (uint64_t)(void *)sa_mad;
>>>> + ssge->length = MAD_BLOCK_SIZE;
>>>> ssge->lkey = cache->mr->lkey;
>>>> }
>>>>
>>>> static int get_pathrecord_info(struct mca_btl_openib_sa_qp_cache *cache,
>>>> - struct ib_mad_sa *sag,
>>>> - struct ib_mad_sa *sar,
>>>> - struct ibv_send_wr *swr,
>>>> - uint16_t lid,
>>>> - uint16_t rem_lid)
>>>> + ib_sa_mad_t *req_mad,
>>>> + ib_sa_mad_t *resp_mad,
>>>> + struct ibv_send_wr *swr,
>>>> + uint16_t lid,
>>>> + uint16_t rem_lid)
>>>> {
>>>> struct ibv_send_wr *bswr;
>>>> struct ibv_wc wc;
>>>> struct timeval get_sl_rec_last_sent, get_sl_rec_last_poll;
>>>> struct ibv_recv_wr *brwr;
>>>> int got_sl_value, get_sl_rec_retries, rc, ne, i;
>>>> + ib_path_rec_t *req_path_record = ib_sa_mad_get_payload_ptr(req_mad);
>>>> + ib_path_rec_t *resp_path_record = ib_sa_mad_get_payload_ptr(resp_mad);
>>>>
>>>> got_sl_value = 0;
>>>> get_sl_rec_retries = 0;
>>>>
>>>> + rc = ibv_post_recv(cache->qp,&(cache->rwr),&brwr);
>>>> + if (0 != rc) {
>>>> + BTL_ERROR(("error posting receive on QP [0x%x] errno says: %s [%d]",
>>>> + cache->qp->qp_num, strerror(errno), errno));
>>>> + return OMPI_ERROR;
>>>> + }
>>>> +
>>>> while (0 == got_sl_value) {
>>>> rc = ibv_post_send(cache->qp, swr,&bswr);
>>>> if (0 != rc) {
>>>> - BTL_ERROR(("error posing send on QP[%x] errno says: %s [%d]",
>>>> + BTL_ERROR(("error posting send on QP [0x%x] errno says: %s [%d]",
>>>> cache->qp->qp_num, strerror(errno), errno));
>>>> return OMPI_ERROR;
>>>> }
>>>> @@ -1190,25 +1157,23 @@
>>>>
>>>> while (0 == got_sl_value) {
>>>> ne = ibv_poll_cq(cache->cq, 1,&wc);
>>>> - if (ne> 0
>>>> -&& wc.status == IBV_WC_SUCCESS
>>>> -&& wc.opcode == IBV_WC_RECV
>>>> -&& wc.byte_len>= sizeof(*sar)
>>>> -&& sar->mad_hdr.tid[0] == sag->mad_hdr.tid[0]
>>>> -&& sar->mad_hdr.tid[1] == sag->mad_hdr.tid[1]) {
>>>> - if (0 == sar->mad_hdr.status
>>>> -&& sar->sa_data.path_record.slid == htons(lid)
>>>> -&& sar->sa_data.path_record.dlid == htons(rem_lid)) {
>>>> + if (ne> 0&&
>>>> + IBV_WC_SUCCESS == wc.status&&
>>>> + IBV_WC_RECV == wc.opcode&&
>>>> + wc.byte_len>= MAD_BLOCK_SIZE&&
>>>> + resp_mad->trans_id == req_mad->trans_id) {
>>>> + if (0 == resp_mad->status&&
>>>> + req_path_record->slid == htons(lid)&&
>>>> + req_path_record->dlid == htons(rem_lid)) {
>>>> /* Everything matches, so we have the desired SL */
>>>> - cache->sl_values[rem_lid] =
>>>> - sar->sa_data.path_record.qos_class_sl& IB_PATH_REC_SL_MASK;
>>>> + cache->sl_values[rem_lid] = ib_path_rec_sl(resp_path_record);
>>>> got_sl_value = 1; /* still must repost recieve buf */
>>>> } else {
>>>> /* Probably bad status, unlikely bad lid match. We will */
>>>> /* ignore response and let it time out so that we do a */
>>>> /* retry, but after a delay. We must make a new TID so */
>>>> /* the SM doesn't see it as the same request. */
>>>> - sag->mad_hdr.tid[1] += 0x10000;
>>>> + req_mad->trans_id += hton64(1);
>>>> }
>>>> rc = ibv_post_recv(cache->qp,&(cache->rwr),&brwr);
>>>> if (0 != rc) {
>>>> @@ -1249,7 +1214,6 @@
>>>> {
>>>> struct ibv_ah_attr aattr;
>>>> struct ibv_port_attr pattr;
>>>> - struct ibv_recv_wr *brwr;
>>>> int rc;
>>>>
>>>> cache->context = ibv_open_device(context_arg->device);
>>>> @@ -1315,16 +1279,10 @@
>>>> cache->rwr.sg_list =&(cache->rsge);
>>>> memset(&(cache->rsge), 0, sizeof(cache->rsge));
>>>> cache->rsge.addr = (uint64_t)(void *)
>>>> - (cache->send_recv_buffer + sizeof(struct ib_mad_sa));
>>>> - cache->rsge.length = sizeof(struct ib_mad_sa) + 40;
>>>> + (cache->send_recv_buffer + MAD_BLOCK_SIZE);
>>>> + cache->rsge.length = MAD_BLOCK_SIZE + 40;
>>>> cache->rsge.lkey = cache->mr->lkey;
>>>>
>>>> - rc = ibv_post_recv(cache->qp,&(cache->rwr),&brwr);
>>>> - if (0 != rc) {
>>>> - BTL_ERROR(("error posing receive on QP[%x] errno says: %s [%d]",
>>>> - cache->qp->qp_num, strerror(errno), errno));
>>>> - return OMPI_ERROR;
>>>> - }
>>>> return 0;
>>>> }
>>>>
>>>> @@ -1334,7 +1292,7 @@
>>>> uint16_t rem_lid)
>>>> {
>>>> struct ibv_send_wr swr;
>>>> - struct ib_mad_sa *sag, *sar;
>>>> + ib_sa_mad_t *req_mad, *resp_mad;
>>>> struct ibv_sge ssge;
>>>> struct mca_btl_openib_sa_qp_cache *cache;
>>>> long page_size = sysconf(_SC_PAGESIZE);
>>>> @@ -1342,8 +1300,8 @@
>>>>
>>>> /* search for a cached item */
>>>> for (cache = sa_qp_cache; cache; cache = cache->next) {
>>>> - if (strcmp(cache->device_name,
>>>> - ibv_get_device_name(context_arg->device)) == 0
>>>> + if (0 == strcmp(cache->device_name,
>>>> + ibv_get_device_name(context_arg->device))
>>>> && cache->port_num == port_num) {
>>>> break;
>>>> }
>>>> @@ -1365,15 +1323,15 @@
>>>>
>>>> /* if the destination lid SL value is not in the cache, go get it */
>>>> if (SL_NOT_PRESENT == cache->sl_values[rem_lid]) {
>>>> - /* sag is first buffer, where we build the SA Get request to send */
>>>> - sag = (struct ib_mad_sa *)(cache->send_recv_buffer);
>>>> + /* sa_mad is first buffer, where we build the SA Get request to send */
>>>> + req_mad = (ib_sa_mad_t *)(cache->send_recv_buffer);
>>>>
>>>> - init_sa_mad(cache, sag,&swr,&ssge, lid, rem_lid);
>>>> + init_sa_mad(cache, req_mad,&swr,&ssge, lid, rem_lid);
>>>>
>>>> - /* sar is the receive buffer (40 byte GRH) */
>>>> - sar = (struct ib_mad_sa *)(cache->send_recv_buffer + sizeof(struct ib_mad_sa) + 40);
>>>> + /* resp_mad is the receive buffer (40 byte offset is for GRH) */
>>>> + resp_mad = (ib_sa_mad_t *)(cache->send_recv_buffer + MAD_BLOCK_SIZE + 40);
>>>>
>>>> - rc = get_pathrecord_info(cache, sag, sar,&swr, lid, rem_lid);
>>>> + rc = get_pathrecord_info(cache, req_mad, resp_mad,&swr, lid, rem_lid);
>>>> if (0 != rc) {
>>>> return rc;
>>>> }
>>>> @@ -1382,3 +1340,4 @@
>>>> /* now all we do is send back the value laying around */
>>>> return cache->sl_values[rem_lid];
>>>> }
>>>> +#endif
>>>> _______________________________________________
>>>> svn-full mailing list
>>>> svn-full_at_[hidden]
>>>> http://www.open-mpi.org/mailman/listinfo.cgi/svn-full
>>>
>>> _______________________________________________
>>> devel mailing list
>>> devel_at_[hidden]
>>> http://www.open-mpi.org/mailman/listinfo.cgi/devel
>>>
>>
>
>