Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: Re: [OMPI users] OpenMPI at scale on Cray XK7
From: Nathan Hjelm (hjelmn_at_[hidden])
Date: 2013-04-24 10:47:12

On Wed, Apr 24, 2013 at 05:01:43PM +0400, Derbunovich Andrei wrote:
> Thank you to everybody for suggestions and comments.
> I have used relatively small number of nodes (4400). It looks like that
> the main issue that I didn't disable dynamic components opening in my
> openmpi build while keeping MPI installation directory on network file
> system. Oh my god!

Yup, that would do it. Cray does static linking of their binaries for a reason. As Ralph suggests take a look at the LANL platform files. You will probably want to compile your app with -static or modify Open MPI's wrapper compilers to always link statically against the libraries needed for MPI. For example

in prefix/share/openmpi/mpicc-wrapper-data.txt (mpicxx, mpif90) modify this line:

to read:
libs=-Wl,-Bstatic libs from libs_static-Wl,-Bdynamic

libs_static=-lmpi -lopen-rte -lopen-pal -lm -lnuma -lpmi -lalpslli -lalpsutil -lugni -lxpmem -lrt -lnsl -lutil

libs=-Wl,-Bstatic -lmpi -lopen-rte -lopen-pal -lm -lnuma -lpmi -lalpslli -lalpsutil -lugni -lxpmem -lrt -lnsl -lutil -Wl,-Bdynamic
libs_static=-lmpi -lopen-rte -lopen-pal -lm -lnuma -lpmi -lalpslli -lalpsutil -lugni -lxpmem -lrt -lnsl -lutil

This will cause your application to link statically against mpi, ugni, xpmem, etc.