Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: Re: [OMPI users] OMPI 1.6.3, InfiniBand and MTL MXM; unable to make it work!
From: Francesco Simula (francesco.simula_at_[hidden])
Date: 2013-01-21 09:26:40


 

Hi Mike,

thanks for the help; yes, the version of MXM has
'centos5u7' in its name, I thought it to be the best match to my Centos
5.8 since there is none signed 'centos5u8' and installing 'centos6u0'
fails complaining of different versions for GLIBC.

The output of
'ibv_devinfo -v' for both machines is:

[cut]

[fsimula_at_agape4 ~]$
sudo /usr/bin/ibv_devinfo -v
hca_id: mlx4_0
 transport: InfiniBand (0)

fw_ver: 2.7.000
 node_guid: 0025:90ff:ff07:0acc
 sys_image_guid:
0025:90ff:ff07:0acf
 vendor_id: 0x02c9
 vendor_part_id: 26428
 hw_ver:
0xB0
 board_id: SM_1061000001000
 phys_port_cnt: 1
 max_mr_size:
0xffffffffffffffff
 page_size_cap: 0xfffffe00
 max_qp: 261824

max_qp_wr: 16351
 device_cap_flags: 0x006c9c76
 max_sge: 32
 max_sge_rd:
0
 max_cq: 65408
 max_cqe: 4194303
 max_mr: 524272
 max_pd: 32764

max_qp_rd_atom: 16
 max_ee_rd_atom: 0
 max_res_rd_atom: 4189184

max_qp_init_rd_atom: 128
 max_ee_init_rd_atom: 0
 atomic_cap: ATOMIC_HCA
(1)
 max_ee: 0
 max_rdd: 0
 max_mw: 0
 max_raw_ipv6_qp: 0

max_raw_ethy_qp: 1
 max_mcast_grp: 8192
 max_mcast_qp_attach: 56

max_total_mcast_qp_attach: 458752
 max_ah: 0
 max_fmr: 0
 max_srq:
65472
 max_srq_wr: 16383
 max_srq_sge: 31
 max_pkeys: 128

local_ca_ack_delay: 15
 port: 1
 state: PORT_ACTIVE (4)
 max_mtu: 2048
(4)
 active_mtu: 2048 (4)
 sm_lid: 4
 port_lid: 8
 port_lmc: 0x00

max_msg_sz: 0x40000000
 port_cap_flags: 0x02510868
 max_vl_num: 8 (4)

bad_pkey_cntr: 0x0
 qkey_viol_cntr: 0x0
 sm_sl: 0
 pkey_tbl_len: 128

gid_tbl_len: 128
 subnet_timeout: 18
 init_type_reply: 0
 active_width:
4X (2)
 active_speed: 10.0 Gbps (4)
 phys_state: LINK_UP (5)
 GID[ 0]:
fe80:0000:0000:0000:0025:90ff:ff07:0acd

[cut]

[cut]

[fsimula_at_agape5 ~]$ sudo /usr/bin/ibv_devinfo -v
hca_id: mlx4_0

transport: InfiniBand (0)
 fw_ver: 2.7.000
 node_guid:
0025:90ff:ff07:0ac4
 sys_image_guid: 0025:90ff:ff07:0ac7
 vendor_id:
0x02c9
 vendor_part_id: 26428
 hw_ver: 0xB0
 board_id: SM_1061000001000

phys_port_cnt: 1
 max_mr_size: 0xffffffffffffffff
 page_size_cap:
0xfffffe00
 max_qp: 261824
 max_qp_wr: 16351
 device_cap_flags:
0x006c9c76
 max_sge: 32
 max_sge_rd: 0
 max_cq: 65408
 max_cqe: 4194303

max_mr: 524272
 max_pd: 32764
 max_qp_rd_atom: 16
 max_ee_rd_atom: 0

max_res_rd_atom: 4189184
 max_qp_init_rd_atom: 128
 max_ee_init_rd_atom:
0
 atomic_cap: ATOMIC_HCA (1)
 max_ee: 0
 max_rdd: 0
 max_mw: 0

max_raw_ipv6_qp: 0
 max_raw_ethy_qp: 1
 max_mcast_grp: 8192

max_mcast_qp_attach: 56
 max_total_mcast_qp_attach: 458752
 max_ah: 0

max_fmr: 0
 max_srq: 65472
 max_srq_wr: 16383
 max_srq_sge: 31

max_pkeys: 128
 local_ca_ack_delay: 15
 port: 1
 state: PORT_ACTIVE (4)

max_mtu: 2048 (4)
 active_mtu: 2048 (4)
 sm_lid: 4
 port_lid: 6

port_lmc: 0x00
 max_msg_sz: 0x40000000
 port_cap_flags: 0x02510868

max_vl_num: 8 (4)
 bad_pkey_cntr: 0x0
 qkey_viol_cntr: 0x0
 sm_sl: 0

pkey_tbl_len: 128
 gid_tbl_len: 128
 subnet_timeout: 18

init_type_reply: 0
 active_width: 4X (2)
 active_speed: 10.0 Gbps (4)

phys_state: LINK_UP (5)
 GID[ 0]:
fe80:0000:0000:0000:0025:90ff:ff07:0ac5

[cut]

Il 2013-01-19 19:12
Mike Dubman ha scritto:

> Hi Francesco,
> Can you please provide
complete output from ibv_devinfo -v command?
> Also, it seems that you
have Centos 5.8 with mxm/centos5.7 installed, will check if there is a
distro version incompatibilities which may cause it and update you.
>

> Alina/Josh - please follow.
>
> Regards
> M
>
> On Thu, Jan 17,
2013 at 4:09 PM, Francesco Simula <francesco.simula_at_[hidden]>
wrote:
>
>> I tried building from OMPI 1.6.3 tarball with the following
./configure:
>> ./configure
--prefix=/apotto/home1/homedirs/fsimula/Lavoro/openmpi-1.6.3/install/

>> --disable-mpi-io
>> --disable-io-romio
>>
--enable-dependency-tracking
>> --without-slurm
>>
--with-platform=optimized
>> --disable-mpi-f77
>> --disable-mpi-f90

>> --with-openib
>> --disable-static
>> --enable-shared
>>
--disable-vt
>> --enable-pty-support
>>
--enable-mca-no-build=btl-ofud,pml-bfo
>> --with-mxm=/opt/mellanox/mxm

>> --with-mxm-libdir=/opt/mellanox/mxm/lib
>>
>> As you can see from
the last two lines, I want to enable the MXM transport layer on a
cluster made of SuperMicro X8DTG-D boards with dual Xeons and Mellanox
MT26428 HCAs; the OS is CentOS 5.8.
>>
>> I tried with two different
.rpm's for MXM, either 'mxm-1.1.ad085ef-1.x86_64-centos5u7.rpm' taken
from here:
>>
http://www.mellanox.com/downloads/hpc/mxm/v1.1/mxm-latest.tar [1]
>>
>>
and 'mxm-1.5.f583875-1.x86_64-centos5u7.rpm' taken from here:
>>
http://www.mellanox.com/downloads/hpc/mxm/v1.5/mxm-latest.tar [2]
>>
>>
With both, even if the compilation concludes successfully, a simple test
(osu_bw from the OSU Micro-Benchmarks 3.8) fails with the sort of
message reported below; the lines:
>>
>> rdma_dev.c:122 MXM DEBUG Port
1 on mlx4_0 has a link layer different from IB. Skipping it
>>
rdma_dev.c:155 MXM ERROR An active IB port on a Mellanox device, with
lid [any] gid [any] not found
>>
>> make it seem like it cannot access
the HW for the HCA: is that so? The very same test works when using
'-mca pml ob1' (thus using the openib BTL).
>>
>> I'm quite ready to
start pulling my hair; any suggestions?
>>
>> The output of
/usr/bin/ibv_devinfo for the two cluster nodes follows:
>> [cut]
>>
hca_id: mlx4_0
>> transport: InfiniBand (0)
>> fw_ver: 2.7.000
>>
node_guid: 0025:90ff:ff07:0ac4
>> sys_image_guid: 0025:90ff:ff07:0ac7
>>
vendor_id: 0x02c9
>> vendor_part_id: 26428
>> hw_ver: 0xB0
>> board_id:
SM_1061000001000
>> phys_port_cnt: 1
>> port: 1
>> state: PORT_ACTIVE
(4)
>> max_mtu: 2048 (4)
>> active_mtu: 2048 (4)
>> sm_lid: 4
>>
port_lid: 6
>> port_lmc: 0x00
>> [/cut]
>>
>> [cut]
>> hca_id:
mlx4_0
>> transport: InfiniBand (0)
>> fw_ver: 2.7.000
>> node_guid:
0025:90ff:ff07:0acc
>> sys_image_guid: 0025:90ff:ff07:0acf
>> vendor_id:
0x02c9
>> vendor_part_id: 26428
>> hw_ver: 0xB0
>> board_id:
SM_1061000001000
>> phys_port_cnt: 1
>> port: 1
>> state: PORT_ACTIVE
(4)
>> max_mtu: 2048 (4)
>> active_mtu: 2048 (4)
>> sm_lid: 4
>>
port_lid: 8
>> port_lmc: 0x00
>> [/cut]
>>
>> The complete output of
the failing test follows:
>>
>> [fsimula_at_agape5
osu-micro-benchmarks-3.8]$ mpirun -x MXM_LOG_LEVEL=poll -mca pml cm -mca
mtl_mxm_np 1 -np 2 -host agape4,agape5
install/libexec/osu-micro-benchmarks/mpi/pt2pt/osu_bw H H
>>
[1358430343.266782] [agape5:8596 :0] config_parser.c:168 MXM DEBUG
>>
[1358430343.266815] [agape5:8596 :0] config_parser.c:168 MXM DEBUG
default: MXM_HANDLE_ERRORS=bt
>> [1358430343.266826] [agape5:8596 :0]
config_parser.c:168 MXM DEBUG default: MXM_GDB_PATH=/usr/bin/gdb
>>
[1358430343.266838] [agape5:8596 :0] config_parser.c:168 MXM DEBUG
default: MXM_DUMP_SIGNO=1
>> [1358430343.266851] [agape5:8596 :0]
config_parser.c:168 MXM DEBUG default: MXM_DUMP_LEVEL=conn
>>
[1358430343.266924] [agape5:8596 :0] config_parser.c:168 MXM DEBUG
default: MXM_ASYNC_MODE=THREAD
>> [1358430343.266936] [agape5:8596 :0]
config_parser.c:168 MXM DEBUG default: MXM_TIME_ACCURACY=0.1
>>
[1358430343.266956] [agape5:8596 :0] config_parser.c:168 MXM DEBUG
default: MXM_PTLS=self,shm,rdma
>> [1358430343.267249] [agape5:8596 :0]
mpool.c:265 MXM DEBUG mpool 'ptl_self_recv_ev': allocated chunk
0xc075f40 of 96016 bytes with 1000 elements
>> [1358430343.267308]
[agape5:8596 :0] mpool.c:156 MXM DEBUG mpool 'ptl_self_recv_ev': align
16, maxelems 1000, elemsize 88, padding 8
>> [1358430343.267316]
[agape5:8596 :0] self.c:410 MXM DEBUG Created ptl_self
>>
[1358430343.267333] [agape5:8596 :0] shm_ptl.c:56 MXM DEBUG Created
ptl_shm
>> [1358430343.268457] [agape5:8596 :0] rdma_ptl.c:65 MXM TRACE
Got 1 IB devices
>> [1358430343.268640] [agape5:8596 :0] rdma_ptl.c:112
MXM DEBUG added device mlx4_0
>> [1358430343.268665] [agape5:8596 :0]
memreg.c:187 MXM TRACE Created memory registration cache on 1 devices
>>
[1358430343.268676] [agape5:8596 :0] rdma_ptl.c:133 MXM DEBUG Created
ptl_rdma
>> [1358430343.268689] [agape5:8596 :0] event.c:353 MXM FUNC
mxm_event_init(event=0x2b73e0ee3038 mode=2 time_accuracy=160000000)
>>
[1358430343.268698] [agape5:8596 :0] timerq.c:55 MXM FUNC
mxm_timerq_init(timerq=0x2b73e0ee3060 accuracy=160000000
max_interval=1600000000)
>> [1358430343.268706] [agape5:8596 :0]
event.c:292 MXM FUNC
mxm_event_add_thread_context(thread=0x2b73e0ee30d0)
>>
[1358430343.268732] [agape5:8596 :0] event.c:198 MXM FUNC
mxm_set_fd_nonblock(fd=10)
>> [1358430343.268741] [agape5:8596 :0]
event.c:198 MXM FUNC mxm_set_fd_nonblock(fd=11)
>> [1358430343.268841]
[agape5:8596 :0] mxm.c:162 MXM INFO context 0x2b73e0ee3010 created
>>
[1358430343.269090] [agape5:8596 :1] event.c:41 MXM FUNC
__call_handler(handler->cb=0x2b73e0ab28a0
handler->arg=0x2b73e0ee3038)
>> [1358430343.269104] [agape5:8596 :1]
timerq.c:88 MXM FUNC mxm_timerq_sweep(timerq=0x2b73e0ee3060
current_time=568595527963578)
>> [1358430343.274685] [agape5:8596 :0]
config_parser.c:168 MXM DEBUG default: MXM_ENABLE_HUGETLB=1
>>
[1358430343.274700] [agape5:8596 :0] config_parser.c:168 MXM DEBUG
default: MXM_ENABLE_TIMEOUTS=y
>> [1358430343.274709] [agape5:8596 :0]
config_parser.c:168 MXM DEBUG default: MXM_ACK_TIMEOUT=0.3
>>
[1358430343.274721] [agape5:8596 :0] config_parser.c:168 MXM DEBUG
default: MXM_POLL_INTERVAL=0.1
>> [1358430343.274742] [agape5:8596 :0]
config_parser.c:168 MXM DEBUG default: MXM_WINDOW_SIZE=512
>>
[1358430343.274755] [agape5:8596 :0] config_parser.c:168 MXM DEBUG
default: MXM_TX_BATCH=1
>> [1358430343.274764] [agape5:8596 :0]
config_parser.c:168 MXM DEBUG default: MXM_CQ_MODERATION=64
>>
[1358430343.274773] [agape5:8596 :0] config_parser.c:168 MXM DEBUG
default: MXM_DRAIN_CQ=n
>> [1358430343.274782] [agape5:8596 :0]
config_parser.c:168 MXM DEBUG default: MXM_RNDV_THRESH=65536
>>
[1358430343.274791] [agape5:8596 :0] config_parser.c:168 MXM DEBUG
default: MXM_ZCOPY_THRESH=2040
>> [1358430343.274815] [agape5:8596 :0]
config_parser.c:168 MXM DEBUG default: MXM_RESIZE_CQ=y
>>
[1358430343.274826] [agape5:8596 :0] config_parser.c:168 MXM DEBUG
default: MXM_UD_MTU=65536
>> [1358430343.274836] [agape5:8596 :0]
config_parser.c:168 MXM DEBUG default: MXM_UD_RX_QUEUE_LEN=16000
>>
[1358430343.274849] [agape5:8596 :0] config_parser.c:168 MXM DEBUG
default: MXM_UD_TX_QUEUE_LEN=64
>> [1358430343.274859] [agape5:8596 :0]
config_parser.c:168 MXM DEBUG default: MXM_UD_RX_MAX_BUFFERS=128000
>>
[1358430343.274877] [agape5:8596 :0] config_parser.c:168 MXM DEBUG
default: MXM_UD_TX_MAX_BUFFERS=8192
>> [1358430343.274887] [agape5:8596
:0] config_parser.c:168 MXM DEBUG default: MXM_UD_RX_DROP_RATE=0
>>
[1358430343.274896] [agape5:8596 :0] config_parser.c:168 MXM DEBUG
default: MXM_UD_ENABLE_NAK=y
>> [1358430343.274904] [agape5:8596 :0]
config_parser.c:168 MXM DEBUG default: MXM_UD_RX_FILL_THRESH=0.6
>>
[1358430343.274915] [agape5:8596 :0] config_parser.c:168 MXM DEBUG
default: MXM_UD_TX_MAX_INLINE=128
>> [1358430343.274925] [agape5:8596
:0] config_parser.c:168 MXM DEBUG default:
MXM_SHM_RX_MAX_BUFFERS=2000
>> [1358430343.274941] [agape5:8596 :0]
config_parser.c:168 MXM DEBUG default: MXM_RDMA_ALLOC=1
>>
[1358430343.274968] [agape5:8596 :0] ep.c:36 MXM FUNC
mxm_ep_create(context=0x2b73e0ee3010)
>> [1358430343.274984]
[agape5:8596 :0] self.c:380 MXM DEBUG Created ptl_self
EP(rank=3767085072)
>> [1358430343.275028] [agape5:8596 :0]
shm_queue.c:230 MXM DEBUG shm_ep=0, shmid=6815750
>> [1358430343.275072]
[agape5:8596 :0] mpool.c:265 MXM DEBUG mpool 'shm_ep_recv': allocated
chunk 0x2aaaadd0c010 of 65824016 bytes with 2000 elements
>>
[1358430343.278550] [agape5:8596 :0] mpool.c:156 MXM DEBUG mpool
'shm_ep_recv': align 16, maxelems 2000, elemsize 32904, padding 8
>>
[1358430343.278584] [agape5:8596 :0] timerq.c:139 MXM FUNC
mxm_timer_schedule(timerq=0x2b73e0ee3060 timer=0xc029538
expiration=568595550657300)
>> [1358430343.278594] [agape5:8596 :0]
timerq.c:43 MXM FUNC mxm_timerq_insert_timer(put timer 0xc029538
expiration 568595550657300 in slot 10)
>> [1358430343.278608]
[agape5:8596 :0] timerq.c:145 MXM TRACE added timer 0xc029538 expiration
568595550657300 interval 160000000
>> [1358430343.278617] [agape5:8596
:0] shm_ep.c:176 MXM DEBUG Created ptl_shm EP (rank=0, ctx_id=1)
>>
[1358430343.278641] [agape5:8596 :0] rdma_ep.c:317 MXM FUNC
mxm_rdma_ep_create()
>> [1358430343.278722] [agape5:8596 :0]
rdma_dev.c:194 MXM FUNC mxm_rdma_dev_init(dev=0xc0b3f00)
>>
[1358430343.278924] [agape5:8596 :0] rdma_dev.c:122 MXM DEBUG Port 1 on
mlx4_0 has a link layer different from IB. Skipping it
>>
[1358430343.278939] [agape5:8596 :0] rdma_dev.c:155 MXM ERROR An active
IB port on a Mellanox device, with lid [any] gid [any] not found
>>
[1358430343.278954] [agape5:8596 :0] timerq.c:150 MXM FUNC
mxm_timer_cancel(timerq=0x2b73e0ee3060 timer=0xc029538)
>>
[1358430343.279454] [agape5:8596 :0] mpool.c:184 MXM DEBUG mpool
'shm_ep_recv': destroyed
>> [1358430343.279466] [agape5:8596 :0]
self.c:287 MXM FUNC mxm_self_ep_destroy(ep=0xc094600)
>>
--------------------------------------------------------------------------
>>
MXM was unable to create an endpoint. Please make sure that the network
link is
>> active on the node and the hardware is functioning.
>>
>>
Error: No such device
>>
>>
--------------------------------------------------------------------------
>>
[1358430343.287336] [agape5:8596 :0] event.c:400 MXM FUNC
mxm_event_cleanup(event=0x2b73e0ee3038)
>> [1358430343.287348]
[agape5:8596 :0] event.c:338 MXM FUNC
mxm_event_remove_thread_context(thread=0x2b73e0ee30d0)
>>
[1358430343.287355] [agape5:8596 :0] event.c:145 MXM FUNC
mxm_event_thread_wakeup()
>> [1358430343.371011] [agape5:8596 :0]
timerq.c:76 MXM FUNC mxm_timerq_cleanup(timerq=0x2b73e0ee3060)
>>
[1358430343.371030] [agape5:8596 :0] memreg.c:194 MXM TRACE Destroying
memory registration cache
>> [1358430343.371129] [agape5:8596 :0]
shm_ptl.c:34 MXM FUNC ptl_shm_destroy(ptl=0xc0729b0)
>>
[1358430343.371139] [agape5:8596 :0] self.c:340 MXM FUNC
mxm_self_destroy(ptl=0xc0699a0)
>> [1358430343.371148] [agape5:8596 :0]
mpool.c:184 MXM DEBUG mpool 'ptl_self_recv_ev': destroyed
>>
[1358430343.371156] [agape5:8596 :0] mxm.c:197 MXM INFO context
0x2b73e0ee3010 destroyed
>>
--------------------------------------------------------------------------
>>
No available pml components were found!
>>
>> This means that there
are no components of this type installed on your
>> system or all the
components reported that they could not be used.
>>
>> This is a fatal
error; your MPI process is likely to abort. Check the
>> output of the
"ompi_info" command and ensure that components of this
>> type are
available on your system. You may also wish to check the
>> value of the
"component_path" MCA parameter and ensure that it has at
>> least one
directory that contains valid MCA components.
>>
--------------------------------------------------------------------------
>>
[agape5:08596] PML cm cannot be selected
>>
--------------------------------------------------------------------------
>>
mpirun has exited due to process rank 1 with PID 8596 on
>> node agape5
exiting improperly. There are two reasons this could occur:
>>
>> 1.
this process did not call "init" before exiting, but others in
>> the
job did. This can cause a job to hang indefinitely while it waits
>> for
all processes to call "init". By rule, if one process calls "init",
>>
then ALL processes must call "init" prior to termination.
>>
>> 2. this
process called "init", but exited without calling "finalize".
>> By
rule, all processes that call "init" MUST call "finalize" prior to
>>
exiting or it will be considered an "abnormal termination"
>>
>> This
may have caused other processes in the application to be
>> terminated
by signals sent by mpirun (as reported here).
>>
--------------------------------------------------------------------------
>>

>> Regards,
>> Francesco
>>
>>
_______________________________________________
>> users mailing list
>>
users_at_[hidden]
>> http://www.open-mpi.org/mailman/listinfo.cgi/users
[3]
 

Links:
------
[1]
http://www.mellanox.com/downloads/hpc/mxm/v1.1/mxm-latest.tar
[2]
http://www.mellanox.com/downloads/hpc/mxm/v1.5/mxm-latest.tar
[3]
http://www.open-mpi.org/mailman/listinfo.cgi/users