Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: Re: [OMPI users] Relocating an Open MPI installation using OPAL_PREFIX
From: Jeff Squyres (jsquyres_at_[hidden])
Date: 2009-01-05 20:02:51


I honestly haven't thought through the ramifications of doing a multi-
lib build with OPAL_PREFIX et al. :-\

If you setenv OPAL_LIBDIR, it'll use whatever you set it to, so it
doesn't matter what you configured --libdir with. Additionally mca/
installdirs/config/install_dirs.h has this by default:

#define OPAL_LIBDIR "${exec_prefix}/lib"

Hence, if you use a default --libdir and setenv OPAL_PREFIX, then the
libdir should pick up the right thing (because it's based on the
prefix). But if you use --libdir that is *not* based on $
{exec_prefix}, then you might run into problems.

Perhaps you can '--libdir="${exec_prefix}/lib64"' so that you can have
your custom libdir, but still have it dependent upon the prefix that
gets expanded at run time...?

(again, I'm not thinking all of this through -- just offering a few
suggestions off the top of my head that you'll need to test / trace
the code to be sure...)

On Jan 5, 2009, at 1:35 PM, Ethan Mallove wrote:

> On Thu, Dec/25/2008 08:12:49AM, Jeff Squyres wrote:
>> It's quite possible that we don't handle this situation properly.
>> Won't
>> you need to libdir's (one for the 32 bit OMPI executables, and one
>> for the
>> 64 bit MPI apps)?
>
> I don't need an OPAL environment variable for the executables, just a
> single OPAL_LIBDIR var for the libraries. (One set of 32-bit
> executables runs with both 32-bit and 64-bit libraries.) I'm guessing
> OPAL_LIBDIR will not work for you if you configure with a non-standard
> --libdir option.
>
> -Ethan
>
>
>>
>> On Dec 23, 2008, at 3:58 PM, Ethan Mallove wrote:
>>
>>> I think the problem is that I am doing a multi-lib build. I have
>>> 32-bit libraries in lib/, and 64-bit libraries in lib/64. I assume I
>>> do not see the issue for 32-bit tests, because all the dependencies
>>> are where Open MPI expects them to be. For the 64-bit case, I tried
>>> setting OPAL_LIBDIR to /opt/openmpi-relocated/lib/lib64, but no
>>> luck.
>>> Given the below configure arguments, what do my OPAL_* env vars need
>>> to be? (Also, could using --enable-orterun-prefix-by-default
>>> interfere
>>> with OPAL_PREFIX?)
>>>
>>> $ ./configure CC=cc CXX=CC F77=f77 FC=f90 --with-openib
>>> --without-udapl --disable-openib-ibcm --enable-heterogeneous
>>> --enable-cxx-exceptions --enable-shared --enable-orterun-prefix-by-
>>> default
>>> --with-sge --enable-mpi-f90 --with-mpi-f90-size=small
>>> --disable-mpi-threads --disable-progress-threads --disable-debug
>>> CFLAGS="-m32 -xO5" CXXFLAGS="-m32 -xO5" FFLAGS="-m32 -xO5"
>>> FCFLAGS="-m32
>>> -xO5"
>>> --prefix=/workspace/em162155/hpc/mtt-scratch/burl-ct-v20z-12/ompi-
>>> tarball-testing/installs/DGQx/install
>>> --mandir=/workspace/em162155/hpc/mtt-scratch/burl-ct-v20z-12/ompi-
>>> tarball-testing/installs/DGQx/install/man
>>> --libdir=/workspace/em162155/hpc/mtt-scratch/burl-ct-v20z-12/ompi-
>>> tarball-testing/installs/DGQx/install/lib
>>> --includedir=/workspace/em162155/hpc/mtt-scratch/burl-ct-v20z-12/
>>> ompi-tarball-testing/installs/DGQx/install/include
>>> --without-mx --with-tm=/ws/ompi-tools/orte/torque/current/shared-
>>> install32
>>> --with-contrib-vt-flags="--prefix=/workspace/em162155/hpc/mtt-
>>> scratch/burl-ct-v!
>>> 20z-12/ompi-tarball-testing/installs/DGQx/install
>>> --mandir=/workspace/em162155/hpc/mtt-scratch/burl-ct-v20z-12/ompi-
>>> tarball-testing/installs/DGQx/install/man
>>> --libdir=/workspace/em162155/hpc/mtt-scratch/burl-ct-v20z-12/ompi-
>>> tarball-testing/installs/DGQx/install/lib
>>> --includedir=/workspace/em162155/hpc/mtt-scratch/burl-ct-v20z-12/
>>> ompi-tarball-testing/installs/DGQx/install/include
>>> LDFLAGS=-R/workspace/em162155/hpc/mtt-scratch/burl-ct-v20z-12/ompi-
>>> tarball-testing/installs/DGQx/install/lib"
>>>
>>> $ ./confgiure CC=cc CXX=CC F77=f77 FC=f90 --with-openib
>>> --without-udapl --disable-openib-ibcm --enable-heterogeneous
>>> --enable-cxx-exceptions --enable-shared --enable-orterun-prefix-by-
>>> default
>>> --with-sge --enable-mpi-f90 --with-mpi-f90-size=small
>>> --disable-mpi-threads --disable-progress-threads --disable-debug
>>> CFLAGS="-m64 -xO5" CXXFLAGS="-m64 -xO5" FFLAGS="-m64 -xO5"
>>> FCFLAGS="-m64
>>> -xO5"
>>> --prefix=/workspace/em162155/hpc/mtt-scratch/burl-ct-v20z-12/ompi-
>>> tarball-testing/installs/DGQx/install
>>> --mandir=/workspace/em162155/hpc/mtt-scratch/burl-ct-v20z-12/ompi-
>>> tarball-testing/installs/DGQx/install/man
>>> --libdir=/workspace/em162155/hpc/mtt-scratch/burl-ct-v20z-12/ompi-
>>> tarball-testing/installs/DGQx/install/lib/lib64
>>> --includedir=/workspace/em162155/hpc/mtt-scratch/burl-ct-v20z-12/
>>> ompi-tarball-testing/installs/DGQx/install/include/64
>>> --without-mx --with-tm=/ws/ompi-tools/orte/torque/current/shared-
>>> install64
>>> --with-contrib-vt-flags="--prefix=/workspace/em162155/hpc/mtt-
>>> scratch/!
>>> burl-ct-v20z-12/ompi-tarball-testing/installs/DGQx/install
>>> --mandir=/workspace/em162155/hpc/mtt-scratch/burl-ct-v20z-12/ompi-
>>> tarball-testing/installs/DGQx/install/man
>>> --libdir=/workspace/em162155/hpc/mtt-scratch/burl-ct-v20z-12/ompi-
>>> tarball-testing/installs/DGQx/install/lib/lib64
>>> --includedir=/workspace/em162155/hpc/mtt-scratch/burl-ct-v20z-12/
>>> ompi-tarball-testing/installs/DGQx/install/include/64
>>> LDFLAGS=-R/workspace/em162155/hpc/mtt-scratch/burl-ct-v20z-12/ompi-
>>> tarball-testing/installs/DGQx/install/lib"
>>> --disable-binaries
>>>
>>> -Ethan
>>>
>>>>
>>>>
>>>> On Dec 22, 2008, at 12:42 PM, Ethan Mallove wrote:
>>>>
>>>>> Can anyone get OPAL_PREFIX to work on Linux? A simple test is to
>>>>> see
>>>>> if the following works for any mpicc/mpirun:
>>>>>
>>>>> $ mv <openmpi-installation> /tmp/foo
>>>>> $ set OPAL_PREFIX /tmp/foo
>>>>> $ mpicc ...
>>>>> $ mpirun ...
>>>>>
>>>>> If you are able to get the above to run successfully, I'm
>>>>> interested
>>>>> in your config.log file.
>>>>>
>>>>> Thanks,
>>>>> Ethan
>>>>>
>>>>>
>>>>> On Thu, Dec/18/2008 11:03:25AM, Ethan Mallove wrote:
>>>>>> Hello,
>>>>>>
>>>>>> The below FAQ lists instructions on how to use a relocated Open
>>>>>> MPI
>>>>>> installation:
>>>>>>
>>>>>> http://www.open-mpi.org/faq/?category=building#installdirs
>>>>>>
>>>>>> On Solaris, OPAL_PREFIX and friends (documented in the FAQ)
>>>>>> work for
>>>>>> me with both MPI (hello_c) and non-MPI (hostname) programs. On
>>>>>> Linux,
>>>>>> I can only get the non-MPI case to work. Here are the environment
>>>>>> variables I am setting:
>>>>>>
>>>>>> $ cat setenv_opal_prefix.csh
>>>>>> set opal_prefix = "/opt/openmpi-relocated"
>>>>>>
>>>>>> setenv OPAL_PREFIX $opal_prefix
>>>>>> setenv OPAL_BINDIR $opal_prefix/bin
>>>>>> setenv OPAL_SBINDIR $opal_prefix/sbin
>>>>>> setenv OPAL_DATAROOTDIR $opal_prefix/share
>>>>>> setenv OPAL_SYSCONFDIR $opal_prefix/etc
>>>>>> setenv OPAL_SHAREDSTATEDIR $opal_prefix/com
>>>>>> setenv OPAL_LOCALSTATEDIR $opal_prefix/var
>>>>>> setenv OPAL_LIBDIR $opal_prefix/lib
>>>>>> setenv OPAL_INCLUDEDIR $opal_prefix/include
>>>>>> setenv OPAL_INFODIR $opal_prefix/info
>>>>>> setenv OPAL_MANDIR $opal_prefix/man
>>>>>>
>>>>>> setenv PATH $opal_prefix/bin:$PATH
>>>>>> setenv LD_LIBRARY_PATH $opal_prefix/lib:$opal_prefix/lib/64
>>>>>>
>>>>>> Here is the error I get:
>>>>>>
>>>>>> $ mpirun -np 2 hello_c
>>>>>>
>>>>>> --------------------------------------------------------------------------
>>>>>> It looks like opal_init failed for some reason; your parallel
>>>>>> process
>>>>>> is
>>>>>> likely to abort. There are many reasons that a parallel
>>>>>> process can
>>>>>> fail during opal_init; some of which are due to configuration or
>>>>>> environment problems. This failure appears to be an internal
>>>>>> failure;
>>>>>> here's some additional information (which may only be relevant
>>>>>> to an
>>>>>> Open MPI developer):
>>>>>>
>>>>>> opal_carto_base_select failed
>>>>>> --> Returned value -13 instead of OPAL_SUCCESS
>>>>>>
>>>>>> --------------------------------------------------------------------------
>>>>>> [burl-ct-v20z-0:27737] [[INVALID],INVALID] ORTE_ERROR_LOG: Not
>>>>>> found in
>>>>>> file runtime/orte_init.c at line 77
>>>>>>
>>>>>> Any ideas on what's going on?
>>>>>>
>>>>>> Thanks,
>>>>>> Ethan
>>>>> _______________________________________________
>>>>> users mailing list
>>>>> users_at_[hidden]
>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>
>>>>
>>>> --
>>>> Jeff Squyres
>>>> Cisco Systems
>>>>
>>>> _______________________________________________
>>>> users mailing list
>>>> users_at_[hidden]
>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>
>>> _______________________________________________
>>> users mailing list
>>> users_at_[hidden]
>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>
>>
>> --
>> Jeff Squyres
>> Cisco Systems
>>
>> _______________________________________________
>> users mailing list
>> users_at_[hidden]
>> http://www.open-mpi.org/mailman/listinfo.cgi/users
> _______________________________________________
> users mailing list
> users_at_[hidden]
> http://www.open-mpi.org/mailman/listinfo.cgi/users

-- 
Jeff Squyres
Cisco Systems