Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: [OMPI users] QLogic HCA random crash after prolonged use
From: Vanja Z (vanja_z_at_[hidden])
Date: 2013-04-20 19:36:24


Hi all,

I have been trying to setup a cluster with QLogic QLE7140 HCA's and a Cisco SFS-7000 24 port switch. The machines are running Debian Wheezy.

I have installed OpenMPI from the repos (1.4.5) and also
libibverbs1
libipathverbs1
libmthca1
librdmacm1
I have also tested OpenMPI compiled from the latest sources (1.6.4) with the same results. I modprobe rdma_ucm, ib_umad and ib_uverbs in order to get MPI jobs to run.

I'm not actually sure if what I've done is enough to correctly configure the network but I have tested several MPI capable codes that use Fortran and C, specifying the openib interface with the flag '--mca btl openib,self'. Things initially work and the bandwidth is as expected, however after anywhere from 4 to 30 hours the jobs crash. The longest job that has completed successfully has gone for around 48 hours however they rarely make it past 4 hours. This is always the error message.

[[36446,1],2][../../../../../../ompi/mca/btl/openib/btl_openib_component.c:3238:handle_wc]
from host84 to: host85 error polling LP CQ with status RETRY EXCEEDED ERROR
status number 12 for wr_id 36085024 opcode 2  vendor error 0 qp_idx 3
--------------------------------------------------------------------------
The OpenFabrics stack has reported a network error event.  Open MPI
will try to continue, but your job may end up failing.

  Local host:        host85
  MPI process PID:   7912
  Error number:      10 (IBV_EVENT_PORT_ERR)

This error may indicate connectivity problems within the fabric;
please contact your system administrator.
--------------------------------------------------------------------------
The InfiniBand retry count between two MPI processes has been
exceeded.  "Retry count" is defined in the InfiniBand spec 1.2
(section 12.7.38):

    The total number of times that the sender wishes the receiver to
    retry timeout, packet sequence, etc. errors before posting a
    completion error.

This error typically means that there is something awry within the
InfiniBand fabric itself.  You should note the hosts on which this
error has occurred; it has been observed that rebooting or removing a
particular host from the job can sometimes resolve this issue.

Two MCA parameters can be used to control Open MPI's behavior with
respect to the retry count:

* btl_openib_ib_retry_count - The number of times the sender will
  attempt to retry (defaulted to 7, the maximum value).
* btl_openib_ib_timeout - The local ACK timeout parameter (defaulted
  to 20).  The actual timeout value used is calculated as:

     4.096 microseconds * (2^btl_openib_ib_timeout)

  See the InfiniBand spec 1.2 (section 12.7.34) for more details.

Below is some information about the host that raised the error and the
peer to which it was connected:

  Local host:   host85
  Local device: qib0
  Peer host:    host84

You may need to consult with your system administrator to get this
problem fixed.
--------------------------------------------------------------------------

On a few occasions the machine that initiated the failure, (host85 in this example) crashed to the point of needing to be power cycled however most times, only Infiniband connectivity was lost after the crash. I have checked kernel and system logs and can't find anything at the time of the crash.

I have seen it recommended to use psm instead of openib for QLogic cards. Could this be part of the problem? After a lot of experimentation I am at a complete loss as to how to get psm up and running. If possible, could someone also help me understand which out of this list (ibibverbs, libipathverbs, libmthca, librdmacm, ib_mad, ib_umad, rdma_ucm, ib_uverbs, ib_qib) is the actual driver for my card and is there any way to configure the driver? This blog posthttp://swik.net/Debian/Planet+Debian/Julien+Blache%3A+QLogic+QLE73xx+InfiniBand+adapters,+QDR,+ib_qib,+OFED+1.5.2+and+Debian+Squeeze/e56if

seems to suggest that I will need to download the complete QLogic OFED stack
and configure the driver which I've tried to do and failed on many
counts.

I would be very grateful for any advice at this stage.

Best regards,

Vanja