It is generally a bad idea to install OMPI in system directories like /usr/local as multiple versions can wind up intermingled with each other, thus causing these kind of problems. I would suggest taking a 1.6.5 tarball and building it with a prefix in your own home directory area. You should then put that location at the *front* of your PATH and LD_LIBRARY_PATH to ensure you get that version instead of something else.

Next, you need to ensure that any launch to a remote node gets pointed to the same location - easiest way to do that is configure OMPI --enable-orterun-prefix-by-default.

That should solve the problem.


On Jan 24, 2014, at 8:40 AM, Elisabeth Beer <elisabethbeer@yahoo.de> wrote:

Hi,

I've done an operating system up-grade to OpenSUSE 13.1 and I've up-graded OpenFOAM from 2.2.1 to 2.2.2.

Bevor, OpenMPI worked well.
Now, it does not work at all.

First Step
------------------
After decomposing the domain, I've tried to start parallel computation:
mpirun -np 8 simpleFoam -parallel

I've received that error message:
It looks like opal_init failed for some reason; your parallel process is
likely to abort. There are many reasons that a parallel process can
fail during opal_init; some of which are due to configuration or
environment problems. This failure appears to be an internal failure;
here's some additional information (which may only be relevant to an
Open MPI developer):

opal_shmem_base_select failed
--> Returned value -1 instead of OPAL_SUCCESS
--------------------------------------------------------------------------
--------------------------------------------------------------------------
It looks like orte_init failed for some reason; your parallel process is
likely to abort. There are many reasons that a parallel process can
fail during orte_init; some of which are due to configuration or
environment problems. This failure appears to be an internal failure;
here's some additional information (which may only be relevant to an
Open MPI developer):

opal_init failed
--> Returned value Error (-1) instead of ORTE_SUCCESS 

I've checked the openmpi version (just to be sure) that was responsible for that error:
mpirun --version
mpirun (Open MPI) 1.7.2

bashrc file of OpenFoam allows to switch between the current systems version an other installed versions.
This version of openmpi (1.7.2) obviously came with OpenSUSE 12.3.
I've received that error with the current systems version 1.7.2
OpenFoam /etc/bashrc/
export WM_MPLIB=SYSTEMOPENMPI

Second Step
-----------------------
I've changed OpenFoams bashrc-file
export WM_MPLIB=OPENMPI

Then I've downloaded ThirdParty2.2.2 package, including openmpi 1.6.3, to see if I could maybe switch and use that (previous) version.

I've received that error message:
mpirun: error while loading shared libraries: libopen-rte.so.4: cannot open shared object file: No such file or directory

I've add another path to LD_LIBRARY_PATH in my home .bashrc including the new path where the missing file is located:
export LD_LIBRARY_PATH=/.../..../usr/local/lib64

Now I get another error:

Sorry! You were supposed to get help about:
opal_init:startup:internal-failure
But I couldn't open the help file:
/opt/ThirdParty-2.2.2/platforms/linux64Gcc/openmpi-1.6.3/share/openmpi/help-opal-runtime.txt: No such file or directory. Sorry!
--------------------------------------------------------------------------
[linux-0ze2:01786] [[INVALID],INVALID] ORTE_ERROR_LOG: Error in file runtime/orte_init.c at line 79
[linux-0ze2:01786] [[INVALID],INVALID] ORTE_ERROR_LOG: Error in file orterun.c at line 694

Third Step
-----------------
I've openmpi 1.6.3 uninstalled with 'make uninstall' and finally downloaded and unzipped openmpi-1.4.5 in /opt/

shell$ cd openmpi-1.4.5
shell$ ./configure --prefix=/usr/local <...lots of output...>
shell$ make all install

I've received following error message:
[linux-0ze2:03493] [[INVALID],INVALID] ORTE_ERROR_LOG: Not found in file runtime/orte_init.c at line 125
--------------------------------------------------------------------------
Sorry!  You were supposed to get help about:
    orte_init:startup:internal-failure
But I couldn't open the help file:
    /usr/local/lib64/openmpi/share/openmpi/help-orte-runtime: No such file or directory.  Sorry!
--------------------------------------------------------------------------
[linux-0ze2:03493] [[INVALID],INVALID] ORTE_ERROR_LOG: Not found in file orterun.c at line 543

I've changed content of PATH variable with help of export PATH=..... in my users .bashrc file

export PATH=/opt/ThirdParty-2.2.2/platforms/linux64Gcc/gperftools-svn/bin:/opt/OpenFOAM-ParaView-3.12.0/bin:/usr/local/lib64/openmpi:/usr/local:/home/lisa/OpenFOAM/lisa-2.2.2/platforms/linux64GccDPOpt/bin:/opt/site/2.2.2/platforms/linux64GccDPOpt/bin:/opt/OpenFOAM-2.2.2/platforms/linux64GccDPOpt/bin:/opt/OpenFOAM-2.2.2/bin:/opt/OpenFOAM-2.2.2/wmake:/home/lisa/bin:/usr/local/bin:/usr/bin:/bin:/usr/bin/X11:/usr/X11R6/bin:/usr/games:/opt/openmpi-1.4.5


The original PATH
/opt/ThirdParty-2.2.2/platforms/linux64Gcc/gperftools-svn/bin:/opt/OpenFOAM-ParaView-3.12.0/bin:/usr/local/lib64/openmpi/bin:/home/lisa/OpenFOAM/lisa-2.2.2/platforms/linux64GccDPOpt/bin:/opt/site/2.2.2/platforms/linux64GccDPOpt/bin:/opt/OpenFOAM-2.2.2/platforms/linux64GccDPOpt/bin:/opt/OpenFOAM-2.2.2/bin:/opt/OpenFOAM-2.2.2/wmake:/home/lisa/bin:/usr/local/bin:/usr/bin:/bin:/usr/bin/X11:/usr/X11R6/bin:/usr/games



I've changed content of LD_LIBRARY_PATH variable with help of export LD_LIBRARY_PATH=..... in my users .bashrc file

export LD_LIBRARY_PATH=/opt/ThirdParty-2.2.2/platforms/linux64Gcc/gperftools-svn/lib:/opt/OpenFOAM-ParaView-3.12.0/lib/paraview-3.12:/opt/OpenFOAM-scotch-6.0.0/lib:/usr/local/lib64/openmpi:/opt/ThirdParty-2.2.2/platforms/linux64GccDPOpt/lib/openmpi-1.6.3:/opt/ThirdParty-2.2.2/platforms/linux64Gcc/openmpi-1.6.3/lib:/opt/ThirdParty-2.2.2/platforms/linux64Gcc/openmpi-1.6.3/lib64:/home/lisa/OpenFOAM/lisa-2.2.2/platforms/linux64GccDPOpt/lib:/opt/site/2.2.2/platforms/linux64GccDPOpt/lib:/opt/OpenFOAM-2.2.2/platforms/linux64GccDPOpt/lib:/opt/ThirdParty-2.2.2/platforms/linux64GccDPOpt/lib:/opt/OpenFOAM-2.2.2/platforms/linux64GccDPOpt/lib/dummy:/usr/lib64/mpi/gcc/openmpi/lib64:/usr/local/lib64:/opt/openmpi-1.4.5:/usr/local

The original LD_LIBRAR_PATH

/opt/ThirdParty-2.2.2/platforms/linux64Gcc/gperftools-svn/lib:/opt/OpenFOAM-ParaView-3.12.0/lib/paraview-3.12:/opt/OpenFOAM-scotch-6.0.0/lib:/opt/OpenFOAM-2.2.2/platforms/linux64GccDPOpt/lib/openmpi-1.4.5:/opt/ThirdParty-2.2.2/platforms/linux64GccDPOpt/lib/openmpi-1.4.5:/usr/local/lib64/openmpi/lib:/usr/local/lib64/openmpi/lib64:/home/lisa/OpenFOAM/lisa-2.2.2/platforms/linux64GccDPOpt/lib:/opt/site/2.2.2/platforms/linux64GccDPOpt/lib:/opt/OpenFOAM-2.2.2/platforms/linux64GccDPOpt/lib:/opt/ThirdParty-2.2.2/platforms/linux64GccDPOpt/lib:/opt/OpenFOAM-2.2.2/platforms/linux64GccDPOpt/lib/dummy:/usr/lib64/mpi/gcc/openmpi/lib64:/usr/local/lib64


I've changed the file /opt/OpenFOAM-2.2.2/etc/config/settings.sh as well! --> OPENMPI

since I've adjusted .bashrc of OpenFoam :

#- MPI implementation:
#    WM_MPLIB = SYSTEMOPENMPI | OPENMPI | MPICH | MPICH-GM | HPMPI
#               | GAMMA | MPI | QSMPI | SGIMPI
export WM_MPLIB=OPENMPI

# Communications library
# ~~~~~~~~~~~~~~~~~~~~~~


/opt/OpenFOAM-2.2.2/etc/config/settings.sh:

unset MPI_ARCH_PATH MPI_HOME FOAM_MPI_LIBBIN

case "$WM_MPLIB" in
SYSTEMOPENMPI)
    # Use the system installed openmpi, get library directory via mpicc
    export FOAM_MPI=openmpi-system

    libDir=`mpicc --showme:link | sed -e 's/.*-L\([^ ]*\).*/\1/'`

    # Bit of a hack: strip off 'lib' and hope this is the path to openmpi
    # include files and libraries.
    export MPI_ARCH_PATH="${libDir%/*}"

    _foamAddLib     $libDir
    unset libDir
    ;;

OPENMPI)
    #export FOAM_MPI=openmpi-1.6.3
    export FOAM_MPI=openmpi-1.4.5
    # optional configuration tweaks:
    _foamSource `$WM_PROJECT_DIR/bin/foamEtcFile config/openmpi.sh`

    #export MPI_ARCH_PATH=$WM_THIRD_PARTY_DIR/platforms/$WM_ARCH$WM_COMPILER/$FOAM_MPI
    export MPI_ARCH_PATH=/usr/local/lib64/openmpi

    # Tell OpenMPI where to find its install directory
    export OPAL_PREFIX=$MPI_ARCH_PATH

    _foamAddPath    $MPI_ARCH_PATH/bin

    # 64-bit on OpenSuSE 12.1 uses lib64 others use lib
    _foamAddLib     $MPI_ARCH_PATH/lib$WM_COMPILER_LIB_ARCH
    _foamAddLib     $MPI_ARCH_PATH/lib

    _foamAddMan     $MPI_ARCH_PATH/share/man
    ;;


I am quite new and don't have experience neither with Linux nor with MPI. Please let me know if I should provide further information.
Please help me to get rid of that error message and make mpi running.

I am working on a single workstation with 16 nodes.



Best regards,
Elisabeth








<config.log.bz2><ompi_info.txt><ompi_info_1.7>_______________________________________________
users mailing list
users@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users