Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |  

This web mail archive is frozen.

This page is part of a frozen web archive of this mailing list.

You can still navigate around this archive, but know that no new mails have been added to it since July of 2016.

Click here to be taken to the new web archives of this list; it includes all the mails that are in this frozen archive plus all new mails that have been sent to the list since it was migrated to the new archives.

From: Emanuel Ziegler (eziegler_at_[hidden])
Date: 2006-02-27 08:29:32


Hi!

After solving my last problem with the help of this list (thanks
again :) I encountered another problem regarding the memory allocation
for the openib component.
If I try to run an arbitrary MPI program, e.g. with
    $ orterun -np 2 --bynode --host node01,node02 \
          --prefix /usr/local ./mpptest -gnuplot
the following error appears:
    [0,1,1][btl_openib.c:787:mca_btl_openib_module_init] error creating
        high priority cq for mthca0 errno says Cannot allocate memory
Obviously, the error occurs only on node02 and not on the local node01
although they are both configured identically. The hosts were cloned
using SystemImager and the problem is symmetric (it always fails on the
remote host).
The FAQ (see
http://www.open-mpi.org/faq/?category=infiniband#ib-locked-pages) blames
that on insufficient user rights for locking memory. So I
adjusted /etc/security/limits.conf and set the hard and soft lilit to
unlimited, but the error remains. The limits are applied correctly as
the command
    $ orterun -np 2 --bynode --host node01,node02 \
          --prefix /usr/local /bin/bash -c ulimit -l
    unlimited
    unlimited
indicated.

The libraries involved are OpenMPI 1.0.2-a7 with libibverbs-1.0-rc5 and
libmthca-1.0-rc5 on Debian sarge with kernel 2.6.15 (from
www.backports.org). There is 8 GB RAM and 16 GB swap available. While
running the program less than 1 GB is used. CQ size is at default
(1000).

Thanks,
  Emanuel