Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: Re: [OMPI users] Passing LD_LIBRARY_PATH to orted
From: Reuti (reuti_at_[hidden])
Date: 2008-10-14 18:03:40


Am 14.10.2008 um 23:39 schrieb Craig Tierney:

> Reuti wrote:
>> Am 14.10.2008 um 23:18 schrieb Craig Tierney:
>>> Ralph Castain wrote:
>>>> I -think- there is...at least here, it does seem to behave that
>>>> way on our systems. Not sure if there is something done locally
>>>> to make it work.
>>>> Also, though, I have noted that LD_LIBRARY_PATH does seem to be
>>>> getting forwarded on the 1.3 branch in some environments. OMPI
>>>> isn't doing it directly to the best of my knowledge, but I think
>>>> the base environment might be. Specifically, I noticed it on
>>>> slurm earlier today. I'll check the others as far as I can.
>>>> Craig: what environment are you using? ssh?
>>>> Ralph
>>>
>>> We are using ssh (we do not use tight integration in SGE).
>> Hi Craig, may I ask why? You compiled Open MPI without SGE
>> support, as in 1.2.7 it's in by default AFAIK? - Reuti
>
> Only because we don't have it on. When we first started using
> SGE around 2002, we hadn't used it. It is on our list of things to

This was still Codine 5.3 - or already SGE?

> do, but it is not trivial to just turn on and validate. We
> compiled all versions
> of OpenMPI we have used (1.2.4,1.2.6, and 1.2.7) with --without-
> gridengine.

It's built-in and you don't need any special start- or
stop_proc_args, just /bin/true will do. It could even be, that
copying the CODINE_* to SGE_* might make Open MPI usable with Codine.

If you want to set some things for ssh login, you can put the
necessary things in ~/.bashrc.

-- Reuti

>
> Craig
>
>
>
>
>
>>>
>>> Craig
>>>
>>>
>>>
>>>
>>>> On Oct 14, 2008, at 1:18 PM, George Bosilca wrote:
>>>>> I use modules too, but they only work locally. Or is there a
>>>>> feature in "module" to automatically load the list of currently
>>>>> loaded local modules remotely ?
>>>>>
>>>>> george.
>>>>>
>>>>> On Oct 14, 2008, at 3:03 PM, Ralph Castain wrote:
>>>>>
>>>>>> You might consider using something like "module" - we use
>>>>>> that system for exactly this reason. Works quite well and
>>>>>> solves the multiple compiler issue.
>>>>>>
>>>>>> Ralph
>>>>>>
>>>>>> On Oct 14, 2008, at 12:56 PM, Craig Tierney wrote:
>>>>>>
>>>>>>> George Bosilca wrote:
>>>>>>>> The option to expand the remote LD_LIBRARY_PATH, in such a
>>>>>>>> way that Open MPI related applications have their
>>>>>>>> dependencies satisfied, is in the trunk. The fact that the
>>>>>>>> compiler requires some LD_LIBRARY_PATH is out of the scope
>>>>>>>> of an MPI implementation, and I don't think we should take
>>>>>>>> care of it.
>>>>>>>> Passing the local LD_LIBRARY_PATH to the remote nodes
>>>>>>>> doesn't make much sense. There are plenty of environment,
>>>>>>>> where the head node have a different configuration than the
>>>>>>>> compute nodes. Again, in this case my original solution
>>>>>>>> seems not that bad. If you copy (or make a link if you
>>>>>>>> prefer) in the Open MPI lib directory to the compiler shared
>>>>>>>> libraries, this will work.
>>>>>>>> george.
>>>>>>>
>>>>>>> This does work. It just increases maintenance for each new
>>>>>>> version
>>>>>>> of OpenMPI. How often does a head node have a different
>>>>>>> configuration
>>>>>>> than the compute node? It would see that this would even
>>>>>>> more support the
>>>>>>> passing of LD_LIBRARY_PATH for OpenMPI tools to support a
>>>>>>> heterogeneous
>>>>>>> configuration as you described.
>>>>>>>
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Craig
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> On Oct 14, 2008, at 12:11 PM, Craig Tierney wrote:
>>>>>>>>> George Bosilca wrote:
>>>>>>>>>> Craig,
>>>>>>>>>> This is a problem with the Intel libraries and not the
>>>>>>>>>> Open MPI ones. You have to somehow make these libraries
>>>>>>>>>> available on the compute nodes.
>>>>>>>>>> What I usually do (but it's not the best way to solve this
>>>>>>>>>> problem) is to copy these libraries somewhere on my home
>>>>>>>>>> area and to add the directory to my LD_LIBRARY_PATH.
>>>>>>>>>> george.
>>>>>>>>>
>>>>>>>>> This is ok when you only ever use one compiler, but it
>>>>>>>>> isn't very flexible.
>>>>>>>>> I want to keep it as simple as possible for my users, while
>>>>>>>>> having a maintainable
>>>>>>>>> system.
>>>>>>>>>
>>>>>>>>> The libraries are on the compute nodes, the problem deals
>>>>>>>>> with supporting
>>>>>>>>> multiple versions of compilers. I can't just list all of
>>>>>>>>> the lib paths
>>>>>>>>> in ld.so.conf, because then the user will never get the
>>>>>>>>> correct one. I can't
>>>>>>>>> specify a static LD_LIBRARY_PATH for the same reason. I
>>>>>>>>> would prefer not
>>>>>>>>> to build my system libraries static.
>>>>>>>>>
>>>>>>>>> To the OpenMPI developers, what is your opinion on changing
>>>>>>>>> orterun/mpirun
>>>>>>>>> to pass LD_LIBRARY_PATH to the remote hosts when starting
>>>>>>>>> OpenMPI processes?
>>>>>>>>> By hand, all that would be done is:
>>>>>>>>>
>>>>>>>>> env LD_LIBRARY_PATH=$LD_LIBRARY_PATH $OPMIPATH/orted <args>
>>>>>>>>>
>>>>>>>>> This would ensure that orted is launched correctly.
>>>>>>>>>
>>>>>>>>> Or is it better to just build the OpenMPI tools
>>>>>>>>> statically? We also
>>>>>>>>> use other compilers (PGI, Lahey) so I need a solution that
>>>>>>>>> works for
>>>>>>>>> all of them.
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Craig
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> On Oct 10, 2008, at 6:17 PM, Craig Tierney wrote:
>>>>>>>>>>> I am having problems launching openmpi jobs on my
>>>>>>>>>>> system. I support multiple versions
>>>>>>>>>>> of MPI and compilers using GNU Modules. For the default
>>>>>>>>>>> compiler, everything is fine.
>>>>>>>>>>> For non-default, I am having problems.
>>>>>>>>>>>
>>>>>>>>>>> I built Openmpi-1.2.6 (and 1.2.7) with the following
>>>>>>>>>>> configure options:
>>>>>>>>>>>
>>>>>>>>>>> # module load intel/10.1
>>>>>>>>>>> # ./configure CC=icc CXX=icpc F77=ifort FC=ifort
>>>>>>>>>>> F90=ifort --prefix=/opt/openmpi/1.2.7-intel-10.1 --without-
>>>>>>>>>>> gridengine --enable-io-romio --with-io-romio-flags=--with-
>>>>>>>>>>> file-sys=nfs+ufs --with-openib=/opt/hjet/ofed/1.3.1
>>>>>>>>>>>
>>>>>>>>>>> When I launch a job, I run the module command for the
>>>>>>>>>>> right compiler/MPI version to set the paths
>>>>>>>>>>> correctly. Mpirun passes LD_LIBRARY_PATH to the
>>>>>>>>>>> executable I am launching, but not orted.
>>>>>>>>>>>
>>>>>>>>>>> When orted is launched on the remote system, the
>>>>>>>>>>> LD_LIBRARY_PATH
>>>>>>>>>>> doesn't come with, and the Intel 10.1 libraries can't be
>>>>>>>>>>> found.
>>>>>>>>>>>
>>>>>>>>>>> /opt/openmpi/1.2.7-intel-10.1/bin/orted: error while
>>>>>>>>>>> loading shared libraries: libintlc.so.5: cannot open
>>>>>>>>>>> shared object file: No such file or directory
>>>>>>>>>>>
>>>>>>>>>>> How do others solve this problem?
>>>>>>>>>>>
>>>>>>>>>>> Thanks,
>>>>>>>>>>> Craig
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Craig Tierney (craig.tierney_at_[hidden])
>>>>>>>>>>> _______________________________________________
>>>>>>>>>>> users mailing list
>>>>>>>>>>> users_at_[hidden]
>>>>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>>>>>>> _______________________________________________
>>>>>>>>>> users mailing list
>>>>>>>>>> users_at_[hidden]
>>>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Craig Tierney (craig.tierney_at_[hidden])
>>>>>>>>> _______________________________________________
>>>>>>>>> users mailing list
>>>>>>>>> users_at_[hidden]
>>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>>>>> _______________________________________________
>>>>>>>> users mailing list
>>>>>>>> users_at_[hidden]
>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Craig Tierney (craig.tierney_at_[hidden])
>>>>>>> _______________________________________________
>>>>>>> users mailing list
>>>>>>> users_at_[hidden]
>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>>>
>>>>>> _______________________________________________
>>>>>> users mailing list
>>>>>> users_at_[hidden]
>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>>
>>>>> _______________________________________________
>>>>> users mailing list
>>>>> users_at_[hidden]
>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>> _______________________________________________
>>>> users mailing list
>>>> users_at_[hidden]
>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>
>>>
>>> --
>>> Craig Tierney (craig.tierney_at_[hidden])
>>> _______________________________________________
>>> users mailing list
>>> users_at_[hidden]
>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>> _______________________________________________
>> users mailing list
>> users_at_[hidden]
>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>
>
> --
> Craig Tierney (craig.tierney_at_[hidden])
> _______________________________________________
> users mailing list
> users_at_[hidden]
> http://www.open-mpi.org/mailman/listinfo.cgi/users