On Wed, 2014-03-12 at 11:50 +0100, Reuti wrote:
> Am 12.03.2014 um 11:39 schrieb Jeff Squyres (jsquyres):
> > Generally, all you need to ensure that your personal copy of OMPI is used is to set the PATH and LD_LIBRARY_PATH to point to your new Open MPI installation. I do this all the time on my development cluster (where I have something like 6 billion different installations of OMPI available... mmm... should probably clean that up...)
> > export LD_LIBRARY_PATH=path_to_my_ompi/lib:$LD_LIBRARY_PATH
> > export PATH=path-to-my-ompi/bin:$PATH
I believe I've already done that. The script the launches everything is
(all one line originally)
PATH=/home/ross/install/bin:$PATH orterun -x R_PROFILE_USER -x
LD_LIBRARY_PATH -x PATH -hostfile ~/KHC/sunbelt/hosts \
-np 7 R --no-save -q
There is a complication with R; it sticks stuff in front of
LD_LIBRARY_PATH. However, the startup script Rmpiprofile fixes that,
though I'm not entirely sure that is effective. However, the old
libraries that are being loaded are not from any directories R added to
LD_LIBRARY_PATH; instead they are from /usr/lib, which is a standard
place for the dynamic loader to look.
> > It should be noted that:
> > 1. you need to *prefix* your PATH and LD_LIBRARY_PATH with these values
> > 2. you need to set these values in a way that will be picked up on all servers that you use in your job. The safest way to do this is in your shell startup files (e.g., $HOME/.bashrc or whatever is relevant for your shell).
> I see "libtorque" in the output below - were these jobs running inside a queuing system? The set paths might be different therein, and need to be set in the job script in this case.
No batch system (see script above for launch mechanism). We threw a lot
of stuff MPI configure was looking for onto the system. AFAIK torque
isn't even installed.
One possible issue is that the Rmpi module for R is not compiled by
mpicc; R has its own notions of proper options for the compiler and its
own infrastructure for building things. I did pass the location of my
local libraries into the build process.
This seems more like an issue with the dynamic loader, or with whatever
system R is using when it loads Rmpi.so.
> -- Reuti
> > See http://www.open-mpi.org/faq/?category=running#run-prereqs, http://www.open-mpi.org/faq/?category=running#adding-ompi-to-path, and http://www.open-mpi.org/faq/?category=running#mpirun-prefix.
> > Note the --prefix option that is described in the 3rd FAQ item I cited -- that can be a bit easier, too.
> > On Mar 12, 2014, at 2:51 AM, Ross Boylan <ross_at_[hidden]> wrote:
> >> I took the advice here and built a personal copy of the current openmpi,
> >> to see if the problems I was having with Rmpi were a result of the old
> >> version on the system.
> >> When I do ldd on the relevant libraries (Rmpi.so is loaded dynamically
> >> by R) everything looks fine; path references that should be local are.
> >> But when I run the program and do lsof it shows that both the system and
> >> personal versions of key libraries are opened.
> >> First, does anyone know which library will actually be used, or how to
> >> tell which library is actually used, in this situation. I'm running on
> >> linux (Debian squeeze)?
> >> Second, it there some way to prevent the wrong/old/sytem libraries from
> >> being loaded?
> >> FWIW I'm still seeing the old misbehavior when I run this way, but, as I
> >> said, I'm really not sure which libraries are being used. Since Rmpi
> >> was built against the new/local ones, I think the fact that it doesn't
> >> crash means I really am using the new ones.
> >> Here are highlights of lsof on the process running R:
> >> COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
> >> R 17634 ross cwd DIR 254,2 12288 150773764 /home/ross/KHC/sunbelt
> >> R 17634 ross rtd DIR 8,1 4096 2 /
> >> R 17634 ross txt REG 8,1 5648 3058294 /usr/lib/R/bin/exec/R
> >> R 17634 ross DEL REG 8,1 2416718 /tmp/openmpi-sessions-ross_at_n100_0/60429/1/shared_mem_pool.n100
> >> R 17634 ross mem REG 8,1 335240 3105336 /usr/lib/openmpi/lib/libopen-pal.so.0.0.0
> >> R 17634 ross mem REG 8,1 304576 3105337 /usr/lib/openmpi/lib/libopen-rte.so.0.0.0
> >> R 17634 ross mem REG 8,1 679992 3105332 /usr/lib/openmpi/lib/libmpi.so.0.0.2
> >> R 17634 ross mem REG 8,1 93936 2967826 /usr/lib/libz.so.18.104.22.168
> >> R 17634 ross mem REG 8,1 10648 3187256 /lib/libutil-2.11.3.so
> >> R 17634 ross mem REG 8,1 32320 2359631 /usr/lib/libpciaccess.so.0.10.8
> >> R 17634 ross mem REG 8,1 33368 2359338 /usr/lib/libnuma.so.1
> >> R 17634 ross mem REG 254,2 979113 152045740 /home/ross/install/lib/libopen-pal.so.6.1.0
> >> R 17634 ross mem REG 8,1 183456 2359592 /usr/lib/libtorque.so.2.0.0
> >> R 17634 ross mem REG 254,2 1058125 152045781 /home/ross/install/lib/libopen-rte.so.7.0.0
> >> R 17634 ross mem REG 8,1 49936 2359341 /usr/lib/libibverbs.so.1.0.0
> >> R 17634 ross mem REG 254,2 2802579 152045867 /home/ross/install/lib/libmpi.so.1.3.0
> >> R 17634 ross mem REG 254,2 106626 152046481 /home/ross/Rlib-3.0.1/Rmpi/libs/Rmpi.so
> >> So libmpi, libopen-pal, and libopen-rte all are opened in two versions and two locations.
> >> Thanks.
> >> Ross Boylan
> >> _______________________________________________
> >> users mailing list
> >> users_at_[hidden]
> >> http://www.open-mpi.org/mailman/listinfo.cgi/users
> > --
> > Jeff Squyres
> > jsquyres_at_[hidden]
> > For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/
> > _______________________________________________
> > users mailing list
> > users_at_[hidden]
> > http://www.open-mpi.org/mailman/listinfo.cgi/users
> users mailing list