Hi,
I am have a problem getting Alternative Path Migration (APM) to work
over the InfiniBand ports on my HCA.
Details on my configuration and the issue I have are below. Please
let me know if you can provide any suggestions or corrections to my
configuration? I will be happy to try other experiments and tests or
provide additional details to debug this problem further.
I have reviewed the Open MPI FAQ and the archive of this mailing list
but I was unable to resolve my problem. There was one thread on
mult-rail fail-over with IB but it did not provide sufficient
information.
Thanks for your help,
Jeremy
Configuration:
MPI version 1.4.3 Bundled with OFED.
I have also tested with MPI version 1.5.4 but the results were the same.
I have 2 machines, each machine has a dual port Mellanox IB HCA
Mellanox MCX354A-FCBT (ConnectX-3 FDR).
I have cabled both ports of each HCA to the same IB Switch (Mellanox SX6036).
What I expected to happen:
I am trying to migrate data transmission between 2 ports of the same HCA.
Start an MPI application. Unplug the fiber cable from Port 1 of an
HCA. Observe that the MPI application continues and data is sent
across Port 2 of the HCA.
However, when I unplug the cable from Port 1 of the IB HCA, the MPI
application hangs and I get the following error messages:
Error 10: IBV_EVENT_PORT_ERR
Error 7: IBV_EVENT_PATH_MIG_ERR
Alternative path migration event reported
Trying to find additional path…
APM: already all ports were used port_num 2 apm_port 2
I've pasted the full verbose error message at the bottom of this email.
I started the MPI application using the following mpirun invocation:
mpirun –np 2 –machinefile machines –mca btl_openib_enable_apm_over_ports 1 demo
What works:
I think that the low level Mellanox IB hardware is working as
expected. The switch, cables and both HCA ports move data OK.
If I don't use the btl_openib_enable_apm_over_ports option then MPI
traffic is evenly spread across both Port 1 and Port 2 while it is
running.
Also, I am able to successfully do fail-over using a bonded device
with IP. For example, if I use netperf to send TCP data over a bonded
IPoIB device I get the expected behavior. When I unplug Port 1,
netperf keeps running and traffic goes over Port 2.
Detailed Error Message:
--------------------------------------------------------------------------
The OpenFabrics stack has reported a network error event. Open MPI
will try to continue, but your job may end up failing.
Local host: bones
MPI process PID: 23115
Error number: 10 (IBV_EVENT_PORT_ERR)
This error may indicate connectivity problems within the fabric;
please contact your system administrator.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
The OpenFabrics stack has reported a network error event. Open MPI
will try to continue, but your job may end up failing.
Local host: bones
MPI process PID: 23115
Error number: 7 (IBV_EVENT_PATH_MIG_ERR)
This error may indicate connectivity problems within the fabric;
please contact your system administrator.
--------------------------------------------------------------------------
[bones][[57528,1],0][btl_openib_async.c:327:btl_openib_async_deviceh]
Alternative path migration event reported
[bones][[57528,1],0][btl_openib_async.c:329:btl_openib_async_deviceh]
Trying to find additional path...
[bones][[57528,1],0][btl_openib_async.c:516:apm_update_port] APM:
already all ports were used port_num 2 apm_port 2
|
![[Flag]](../../../../../community/mirrors/flag-us.gif){kind=small}