Josh and I spent some quality time on the phone today talking through
this issue. Here's some points (not in strict order):
- "--enable-static --enable-shared" (either explicit or implicit)
works fine -- it'll build both static and shared library versions of
OMPI's libraries. Unless otherwise specified, the OMPI executables
will link against the shared libraries, but Libtool typically -
rpath's in where the libraries build, so there's no need to set
LD_LIBRARY_PATH to run them.
- It turns out that there is a critical app at Sandia that *requires*
-static to build (those crazy app writers!). Sandia does have all
the right .so libraries installed on all their nodes, so the warnings
about dlopen() and gethostname() are all ok.
- Sandia also does have all the Right .a libraries installed (e.g.,
libsysfs.a, libdl.a, libc.a, libtorque.a, libibverbs.a, ...etc.). So
everything *should* compile properly to get a fully static executable.
- Because horrid details that no one cares about, you can't compile
MPI apps with -static *and* have Open MPI have an internal memory
manager. So it is necessary to configure OMPI with --without-memory-
manager. This is actually ok in OS-bypass network environments
because OMPI defaults to never returning NIC-registered memory to the
user (and therefore you don't need a memory manager). If you want a
memory manager, then you can't compile your MPI apps with -static;
sorry, this is a limitation of Linux.
--> Configuring OMPI with --without-memory-manager resolved all the
duplicate symbol problems.
- OMPI is not correctly determining that libsysfs is a dependency (I
think because it's an implicit dependency from libibverbs). Hence,
compiling MPI applications with the wrapper results in missing
symbols because the wrapper is not adding -lsysfs to the command
line. The workaround is to edit $prefix/share/openmpi/
mpi*wrapper.txt to add "-lsysfs" to the "libs" field (these files are
what the wrapper compilers read to know how to build command lines
for the underlying compiler).
--> Hand-editing mpi*wrapper.txt to insert "-lsysfs" fixed the
missing symbols problem.
End of the day: Sandia was able to get a 100% static MPI application
executable, although ld still emits a bunch of warnings (which are ok).
This is probably FAQ fodder; I have some time on airplanes coming
up. I'll morph this material into stuff that is suitable for the FAQ.
On Oct 16, 2006, at 2:03 AM, Josh England wrote:
> On Sun, 2006-10-15 at 23:31 -0600, Josh England wrote:
>> On Sat, 2006-10-14 at 09:35 -0400, Jeff Squyres wrote:
>>> Given the messages from the linker and my prior mail, I'm not sure
>>> that a) is possible -- it will require more poking around to know
>>> sure. b) is definitely possible simply by using --enable-static and
>>> and then using "mpicc" (and friends) as normal.
>>> OMPI will be built statically (to include torque and libibverbs,
>>> assuming you have .a's versions of both of those), but as Scott
>>> mentioned, system libraries such as dl and sysfs will be linked
>> OK. I've got a build with --enable-static and --disable-shared,
>> and I'm
>> still seeing the *exact* same behavior. I cannot build a static
>> either with mpicc or specifying a manual link line. Note that this
>> appears to be a problem only with 1.1.2_rc4. Open MPI version 1.1.1
>> seems to work.
> Sorry. I wrote too fast. Version 1.1.1 actually *does not* work for
> static builds either. It exhibits the same problem described in the
> original post.
> users mailing list
Server Virtualization Business Unit