Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: [OMPI users] can't use "--mca btl_openib_cpc_include xoob"
From: ãÛÊ¥ÕÜ (ksz_at_[hidden])
Date: 2011-01-01 01:30:56


Hi,
my system have mellanox connectX adapter in all nodes, when use cmd "/sbin/lspci", will print "0c:00.0 InfiniBand: Mellanox Technologies MT25418 [ConnectX IB DDR] (rev a0)". And ofed 1.4.2 has been installed.
I install openmpi 1.4.3 using cmd "./configure --prefix=/home_soft/home/scksz/mpi/ompi143; make all install", the config.log contain those information about openib, xrc or xoob:

<<<<<<<<config.log information about openib<<<<<<<<
configure:5726: checking whether to add padding to the openib control header
configure:5739: result: no
| #define OMPI_OPENIB_PAD_HDR 0
...
configure:126382: checking for m4 configure components in framework btl
configure:126384: result: elan, gm, mx, ofud, openib, portals, tcp, udapl
configure:132447: checking --with-openib value
configure:132450: result: simple ok (unspecified)
configure:132492: checking --with-openib-libdir value
configure:132495: result: simple ok (unspecified)
configure:135339: checking for MCA component btl:openib compile mode
configure:135345: result: dso
configure:135446: checking --with-openib value
configure:135449: result: simple ok (unspecified)
configure:135491: checking --with-openib-libdir value
configure:135494: result: simple ok (unspecified)
configure:138177: checking which openib btl cpcs will be built
configure:138179: result: oob xoob rdmacm
configure:138242: checking if MCA component btl:openib can compile
configure:138244: result: yes
config.status:2748: creating ompi/mca/btl/openib/Makefile
MCA_btl_ALL_COMPONENTS=' self sm elan gm mx ofud openib portals tcp udapl'
MCA_btl_ALL_SUBDIRS=' mca/btl/self mca/btl/sm mca/btl/elan mca/btl/gm mca/btl/mx mca/btl/ofud mca/btl/openib mca/btl/portals mca/btl/tcp mca/btl/udapl'
MCA_btl_DSO_COMPONENTS=' self sm ofud openib tcp'
MCA_btl_DSO_SUBDIRS=' mca/btl/self mca/btl/sm mca/btl/ofud mca/btl/openib mca/btl/tcp'
MCA_btl_openib_have_ibcm_FALSE=''
MCA_btl_openib_have_ibcm_TRUE='#'
MCA_btl_openib_have_rdmacm_FALSE='#'
MCA_btl_openib_have_rdmacm_TRUE=''
MCA_btl_openib_have_xrc_FALSE='#'
MCA_btl_openib_have_xrc_TRUE=''
OMPI_BUILD_btl_openib_DSO_FALSE='#'
OMPI_BUILD_btl_openib_DSO_TRUE=''
btl_openib_CFLAGS=''
btl_openib_CPPFLAGS=''
btl_openib_LDFLAGS=''
btl_openib_LIBS='-lrdmacm -libverbs '
<<<<<<<<config.log information about xrc<<<<<<<<
configure:134262: checking for ibv_create_xrc_rcv_qp
configure:134318: gcc -o conftest -O3 -DNDEBUG -finline-functions -fno-strict-aliasing -pthread conftest.c -lnsl -lutil -lm -libverbs >&5
configure:134325: $? = 0
configure:134347: result: yes
configure:134948: checking if ConnectX XRC support is enabled
configure:134956: result: yes
configure:137261: checking for ibv_create_xrc_rcv_qp
configure:137346: result: yes
configure:137947: checking if ConnectX XRC support is enabled
configure:137955: result: yes
| #define HAVE_IBV_CREATE_XRC_RCV_QP 1
...
| #define OMPI_HAVE_CONNECTX_XRC 1
...
ac_cv_func_ibv_create_xrc_rcv_qp=yes
MCA_btl_openib_have_xrc_FALSE='#'
MCA_btl_openib_have_xrc_TRUE=''
<<<<<<<<config.log information about xoob<<<<<<<<
configure:138179: result: oob xoob rdmacm
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<

after ompi's installation I add the ompi's PATH and LD_LIBRARY_PATH in .bashrc.
Then I compile a mpi code named "mpisample.c", use cmd "mpicc mpisample.c -o a_user.out".
When I run the programm "run a_user.out" using cmd "mpirun -host LB270210,CB060106,CB060107 -np 3 --mca btl_openib_cpc_include xoob --mca orte_base_help_aggregate 0 a_user.out", will print:

<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
--------------------------------------------------------------------------
No OpenFabrics connection schemes reported that they were able to be
used on a specific port. As such, the openib BTL (OpenFabrics
support) will be disabled for this port.

  Local host: CB060107
  Local device: mlx4_0
  Local port: 1
  CPCs attempted: xoob
--------------------------------------------------------------------------
--------------------------------------------------------------------------
No OpenFabrics connection schemes reported that they were able to be
used on a specific port. As such, the openib BTL (OpenFabrics
support) will be disabled for this port.

  Local host: LB270210
  Local device: mlx4_0
  Local port: 1
  CPCs attempted: xoob
--------------------------------------------------------------------------
--------------------------------------------------------------------------
No OpenFabrics connection schemes reported that they were able to be
used on a specific port. As such, the openib BTL (OpenFabrics
support) will be disabled for this port.

  Local host: CB060106
  Local device: mlx4_0
  Local port: 1
  CPCs attempted: xoob
--------------------------------------------------------------------------
Hello! My rank is 0. My hostname is "LB270210"
Hello! My rank is 2. My hostname is "CB060107"
Hello! My rank is 1. My hostname is "CB060106"
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<

When I use cmd "mpirun -host LB270210,CB060106,CB060107 -np 3 --mca btl openib,self,sm a_user.out" or "mpirun -host LB270210,CB060106,CB060107 -np 3 --mca btl openib,self,sm --mca btl_openib_cpc_include rdmacm a_user.out", will print correct result, but use cmd "mpirun -host LB270210,CB060106,CB060107 -np 3 --mca btl openib a_user.out", will print error mesg.

I want to know:
1) When I want to use XRC, I must have a special IB switch?
2) How can I use XRC in ompi,and on which situation the XRC feature will bring benifit?
3) If this is only way to using XRC that using "-mca btl_openib_cpc_include xoob"?
4) When using xoob, which transport and network protocol ompi will use?
5) When using Qlogic adapter, how to optimize opmi?
6) What's mean of "CPC" in ompi?