Open MPI logo

Open MPI Development Mailing List Archives

  |   Home   |   Support   |   FAQ   |  

This web mail archive is frozen.

This page is part of a frozen web archive of this mailing list.

You can still navigate around this archive, but know that no new mails have been added to it since July of 2016.

Click here to be taken to the new web archives of this list; it includes all the mails that are in this frozen archive plus all new mails that have been sent to the list since it was migrated to the new archives.

Subject: Re: [OMPI devel] [OMPI svn-full] svn:open-mpi r24395
From: Jeff Squyres (jsquyres_at_[hidden])
Date: 2011-02-16 08:15:55


Oracle --

Is this really only specific to Solaris? More comments below about configure.m4.

On Feb 16, 2011, at 12:37 AM, dkerr_at_[hidden] wrote:

> Author: dkerr
> Date: 2011-02-16 00:37:22 EST (Wed, 16 Feb 2011)
> New Revision: 24395
> URL: https://svn.open-mpi.org/trac/ompi/changeset/24395
>
> Log:
> on Solaris, when IBV_ACCESS_SO is available, use strong ordered memory region for eager rdma connection
> Text files modified:
> trunk/ompi/mca/btl/openib/btl_openib_component.c | 13 ++++++++++---
> trunk/ompi/mca/btl/openib/btl_openib_endpoint.c | 19 +++++++++++++++++--
> trunk/ompi/mca/btl/openib/configure.m4 | 16 +++++++++++++++-
> 3 files changed, 42 insertions(+), 6 deletions(-)
>
> Modified: trunk/ompi/mca/btl/openib/btl_openib_component.c
> ==============================================================================
> --- trunk/ompi/mca/btl/openib/btl_openib_component.c (original)
> +++ trunk/ompi/mca/btl/openib/btl_openib_component.c 2011-02-16 00:37:22 EST (Wed, 16 Feb 2011)
> @@ -15,7 +15,7 @@
> * Copyright (c) 2006-2007 Los Alamos National Security, LLC. All rights
> * reserved.
> * Copyright (c) 2006-2007 Voltaire All rights reserved.
> - * Copyright (c) 2009-2010 Oracle and/or its affiliates. All rights reserved.
> + * Copyright (c) 2009-2011 Oracle and/or its affiliates. All rights reserved.
> * $COPYRIGHT$
> *
> * Additional copyrights may follow
> @@ -527,9 +527,16 @@
> {
> mca_btl_openib_device_t *device = (mca_btl_openib_device_t*)reg_data;
> mca_btl_openib_reg_t *openib_reg = (mca_btl_openib_reg_t*)reg;
> + enum ibv_access_flags access_flag = IBV_ACCESS_LOCAL_WRITE |
> + IBV_ACCESS_REMOTE_WRITE | IBV_ACCESS_REMOTE_READ;
>
> - openib_reg->mr = ibv_reg_mr(device->ib_pd, base, size, IBV_ACCESS_LOCAL_WRITE |
> - IBV_ACCESS_REMOTE_WRITE | IBV_ACCESS_REMOTE_READ);
> +#if defined(HAVE_IBV_ACCESS_SO)
> + if (reg->flags & MCA_MPOOL_FLAGS_SO_MEM) {
> + access_flag |= IBV_ACCESS_SO;
> + }
> +#endif
> +
> + openib_reg->mr = ibv_reg_mr(device->ib_pd, base, size, access_flag);
>
> if (NULL == openib_reg->mr) {
> return OMPI_ERR_OUT_OF_RESOURCE;
>
> Modified: trunk/ompi/mca/btl/openib/btl_openib_endpoint.c
> ==============================================================================
> --- trunk/ompi/mca/btl/openib/btl_openib_endpoint.c (original)
> +++ trunk/ompi/mca/btl/openib/btl_openib_endpoint.c 2011-02-16 00:37:22 EST (Wed, 16 Feb 2011)
> @@ -16,7 +16,7 @@
> * Copyright (c) 2006-2007 Voltaire All rights reserved.
> * Copyright (c) 2006-2009 Mellanox Technologies, Inc. All rights reserved.
> * Copyright (c) 2010 IBM Corporation. All rights reserved.
> - * Copyright (c) 2010 Oracle and/or its affiliates. All rights reserved
> + * Copyright (c) 2010-2011 Oracle and/or its affiliates. All rights reserved
> *
> * $COPYRIGHT$
> *
> @@ -911,6 +911,7 @@
> char *buf;
> mca_btl_openib_recv_frag_t *headers_buf;
> int i;
> + uint32_t flag = MCA_MPOOL_FLAGS_CACHE_BYPASS;
>
> /* Set local rdma pointer to 1 temporarily so other threads will not try
> * to enter the function */
> @@ -925,11 +926,25 @@
> if(NULL == headers_buf)
> goto unlock_rdma_local;
>
> +#if defined(HAVE_IBV_ACCESS_SO)
> + /* Solaris implements the Relaxed Ordering feature defined in the
> + PCI Specification. With this in mind any memory region which
> + relies on a buffer being written in a specific order, for
> + example the eager rdma connections created in this routinue,
> + must set a strong order flag when registering the memory for
> + rdma operations.
> +
> + The following flag will be interpreted and the appropriate
> + steps will be taken when the memory is registered in
> + openib_reg_mr(). */
> + flag |= MCA_MPOOL_FLAGS_SO_MEM;
> +#endif
> +
> buf = (char *) openib_btl->super.btl_mpool->mpool_alloc(openib_btl->super.btl_mpool,
> openib_btl->eager_rdma_frag_size *
> mca_btl_openib_component.eager_rdma_num,
> mca_btl_openib_component.buffer_alignment,
> - MCA_MPOOL_FLAGS_CACHE_BYPASS,
> + flag,
> (mca_mpool_base_registration_t**)&endpoint->eager_rdma_local.reg);
>
> if(!buf)
>
> Modified: trunk/ompi/mca/btl/openib/configure.m4
> ==============================================================================
> --- trunk/ompi/mca/btl/openib/configure.m4 (original)
> +++ trunk/ompi/mca/btl/openib/configure.m4 2011-02-16 00:37:22 EST (Wed, 16 Feb 2011)
> @@ -12,6 +12,7 @@
> # All rights reserved.
> # Copyright (c) 2007-2010 Cisco Systems, Inc. All rights reserved.
> # Copyright (c) 2008 Mellanox Technologies. All rights reserved.
> +# Copyright (c) 2011 Oracle and/or its affiliates. All rights reserved.
> # $COPYRIGHT$
> #
> # Additional copyrights may follow
> @@ -34,7 +35,7 @@
> AC_DEFUN([MCA_ompi_btl_openib_CONFIG],[
> AC_CONFIG_FILES([ompi/mca/btl/openib/Makefile])
>
> - OPAL_VAR_SCOPE_PUSH([cpcs have_threads])
> + OPAL_VAR_SCOPE_PUSH([cpcs have_threads have_ibv_access_so])
> cpcs="oob"
>
> OMPI_CHECK_OPENIB([btl_openib],
> @@ -78,6 +79,19 @@
> AC_MSG_CHECKING([which openib btl cpcs will be built])
> AC_MSG_RESULT([$cpcs])])
>
> + # check for Solaris specific memory access flag
> + AS_IF([test "$btl_openib_happy" = "yes"],
> + [if test "`echo $build_os | $GREP solaris`"; then

FWIW, the above could be a single AS_IF:

    AS_IF([test "$btl_openib_happy" = "yes" -a -n "echo $build_os | $GREP solaris" ],

> + AC_TRY_COMPILE([#include <infiniband/verbs.h>],
> + [int flag = IBV_ACCESS_SO;],
> + [have_ibv_access_so="yes"
> + AC_DEFINE_UNQUOTED([HAVE_IBV_ACCESS_SO],
> + 1,[openib define HAVE_IBV_ACCESS_SO])],
> + [have_ibv_access_so="no"])
> + AC_MSG_CHECKING([for IBV_ACCESS_SO in Solaris])
> + AC_MSG_RESULT([$have_ibv_access_so])

I *believe* you should be able to replace the entire above section with an AC_CHECK_DECLS. Check elsewhere in ompi_check_openib.m4 for uses of AC_CHECK_DECLS looking for the presence of various IBV_* constants. See:

http://www.gnu.org/software/hello/manual/autoconf/Generic-Declarations.html

> + fi])
> +
> # Enable openib device failover. It is disabled by default.
> AC_ARG_ENABLE([btl-openib-failover],
> [AC_HELP_STRING([--enable-btl-openib-failover],
> _______________________________________________
> svn-full mailing list
> svn-full_at_[hidden]
> http://www.open-mpi.org/mailman/listinfo.cgi/svn-full

-- 
Jeff Squyres
jsquyres_at_[hidden]
For corporate legal information go to:
http://www.cisco.com/web/about/doing_business/legal/cri/