Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

From: Jeff Squyres (jsquyres_at_[hidden])
Date: 2007-10-25 07:54:38


On Oct 24, 2007, at 10:05 PM, Dirk Eddelbuettel wrote:

> | > | If I had to guess, the systems where you don't see the
> warning are
> | > | systems that have OFED loaded.
> | >
> | > I am pretty sure that none of the systems (at work) have IB
> | > hardware. I am
> | > very sure that my home systems do not, and there the 'btl =
> ^openib'
> | > successfully suppresses the warning --- whereas at work it
> doesn't.
> |
> | Note that you don't need to have IB hardware -- all you need is the
> | OFED software loaded. I don't know if Debian ships the OFED
> | libraries by default...? In particular, look for libibverbs:
> |
> | [18:28] svbu-mpi:~/svn/ompi % ldd $bogus/lib/openmpi/
> mca_btl_openib.so
> | libibverbs.so.1 => /usr/lib64/libibverbs.so.1
> | (0x0000002a956c2000)
> | libnsl.so.1 => /lib64/libnsl.so.1 (0x0000002a957cd000)
> | libutil.so.1 => /lib64/libutil.so.1 (0x0000002a958e4000)
> | libm.so.6 => /lib64/tls/libm.so.6 (0x0000002a959e8000)
> | libpthread.so.0 => /lib64/tls/libpthread.so.0
> | (0x0000002a95b6e000)
> | libc.so.6 => /lib64/tls/libc.so.6 (0x0000002a95c83000)
> | libdl.so.2 => /lib64/libdl.so.2 (0x0000002a95eb8000)
> | /lib64/ld-linux-x86-64.so.2 (0x000000552aaaa000)
>
> Good point. However, I use the .deb packages which are I build for
> Debian,
> and they use libibverbs where available:
>
> Build-Depends: [...], libibverbs-dev [!kfreebsd-i386 !kfreebsd-
> amd64 \
> !hurd-i386], gfortran, libsysfs-dev, automake, gcc (>= 4:4.1.2)
>
> in particular on i386. Consequently, the binary package ends up with a
> Depends on the run-time package 'libibverbs1' -- and this will
> hence always
> be present as all my systems use the .deb packages (either from
> Debian or
> locally rebuild) that forces libibverbs1 in via this Depends.
>
> At work, I re-build these same package under Ubuntu on my "head
> node". And
> on the head node, no warning is seen -- wherease my computes issue the
> warning.
>
> Could this be another one of the dlopen issues where basically
> ldopen("libibverbs.so")
> is executed? Because the compute nodes do NOT have libibverbs.so
> (from the
> -dev package) but only libibverbs.so.1.0.0 and its matching symlink
> libibverbs.so.1.

We will not dlopen libibverbs.so directly -- we will only dlopen the
mca_btl_openib.so file. The dynamic linker will automatically open
all of its dependencies. If those dependencies cannot be found /
symbols cannot be resolved, the dynamic linker will fail the dlopen
of libibverbs.

Can you run "ldd mca_btl_openib.so" on your head node and your
compute nodes? See if there's a difference in the output. I think
this is the next step in this troubleshooting process...

> I just tested that hypothesis and install libibverbs-dev, but no
> beans. Still
> get the warning.
>
> | However, I note something in your last reply that I may have missed
> | before -- can you clarify a point for me: are you saying that on
> your
> | home machine, this generates the openib "file not found" warning:
> |
> | mpirun -np 2 hello
> |
> | but this does not:
> |
> | mpirun -np 2 --mca btl ^openib hello
>
> More or less, but I use /etc/openmpi/openmci-mca-params.conf to toggle
> ^openib. Adding it again as --mca btl ^openib changes nothing,
> unfortunately.

This MCA behavior is as expected; adding a param to openmpi-mca-
params.conf is exactly the same as putting it on the command line
(except that the command line has higher precedence).

-- 
Jeff Squyres
Cisco Systems