Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: [OMPI users] Qlogic & openmpi
From: Arnaud Heritier (gmarns_at_[hidden])
Date: 2011-11-27 15:11:42


Hello,

I run into a stange problem with qlogic OFED and openmpi. When i submit
(through SGE) 2 jobs on the same node, the second job ends up with:

(ipath/PSM)[10292]: can't open /dev/ipath, network down (err=26)

I'm pretty sure the infiniband is working well as the other job runs fine.

Here is details about the configuration:

Qlogic HCA: InfiniPath_QMH7342 (2 ports but only one connected to a switch)
qlogic_ofed-1.5.3-7.0.0.0.35 (rocks cluster roll)
openmpi 1.5.4 (./configure --with-psm --with-openib --with-sge)

-------------

In order to fix this problem i recompiled openmpi without psm support, but
i faced an other problem:

The OpenFabrics (openib) BTL failed to initialize while trying to
allocate some locked memory. This typically can indicate that the
memlock limits are set too low. For most HPC installations, the
memlock limits should be set to "unlimited". The failure occured
here:

  Local host: compute-0-6.local
  OMPI source: btl_openib.c:329
  Function: ibv_create_srq()
  Device: qib0
  Memlock limit: *unlimited*