Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: Re: [OMPI users] MPI orte_init fails on remote nodes
From: Richard Bardwell (richard_at_[hidden])
Date: 2012-02-14 06:57:26


OK, I get it running if I specify /usr/local/bin/mpiexec instead of just mpiexec.
Now, the program hangs at the first MPI_Waitall on the remote node.
The program runs just fine if both nodes are on the same machine.

Any ideas how to debug this ?

Many Thanks

Richard
----- Original Message -----
From: "Jeff Squyres" <jsquyres_at_[hidden]>
To: "Open MPI Users" <users_at_[hidden]>
Sent: Tuesday, February 14, 2012 11:13 AM
Subject: Re: [OMPI users] MPI orte_init fails on remote nodes

> Make sure that your LD_LIBRARY_PATH is being set in your shell startup files for *non-interactive logins*.
>
> For example, ensure that LD_LIBRARY_PATH is set properly, even in this case:
>
> -----
> ssh some-other-node env | grep LD_LIBRARY_PATH
> -----
>
> (note that this is different than "ssh some-other-node echo $LD_LIBRARY_PATH", because the "$LD_LIBRARY_PATH" will be evaluated on
> the local node, even before ssh is invoked)
>
> I mention this because some shell startup files distinguish between interactive and non-interactive logins; they sometimes
> terminate early for non-interactive logins. Look for "exit" statements, or conditional blocks that are only invoked during
> interactive logins, for example.
>
>
>
> On Feb 14, 2012, at 5:40 AM, Richard Bardwell wrote:
>
>> Jeff,
>>
>> I wiped out all versions of openmpi on all the nodes including the distro installed version.
>> I reinstalled version 1.4.4 on all nodes.
>> I now get the error that libopen-rte.so.0 cannot be found when running mpiexec across
>> different nodes, even though the LD_LIBRARY_PATH for all nodes points to /usr/local/lib
>> where the file exists. Any ideas ?
>>
>> Many Thanks
>>
>> Richard
>>
>> ----- Original Message ----- From: "Jeff Squyres" <jsquyres_at_[hidden]>
>> To: "Open MPI Users" <users_at_[hidden]>
>> Sent: Monday, February 13, 2012 6:28 PM
>> Subject: Re: [OMPI users] MPI orte_init fails on remote nodes
>>
>>
>>> You might want to fully uninstall the disto-installed version of Open MPI on all the nodes (e.g., Red Hat may have installed a
>>> different version of Open MPI, and that version is being found in your $PATH before your custom-installedversion).
>>>
>>>
>>> On Feb 13, 2012, at 12:12 PM, Richard Bardwell wrote:
>>>
>>>> OK, 1.4.4 is happily installed on both machines. But, I now get a really
>>>> weird error when running on the 2 nodes. I get
>>>> Error: unknown option "--daemonize"
>>>> even though I am just running with -np 2 -hostfile test.hst
>>>>
>>>> The program runs fine on 2 cores if running locally on each node.
>>>>
>>>> Any ideas ??
>>>>
>>>> Thanks
>>>>
>>>> Richard
>>>> ----- Original Message ----- From: "Gustavo Correa" <gus_at_[hidden]>
>>>> To: "Open MPI Users" <users_at_[hidden]>
>>>> Sent: Monday, February 13, 2012 4:22 PM
>>>> Subject: Re: [OMPI users] MPI orte_init fails on remote nodes
>>>>
>>>>
>>>>> On Feb 13, 2012, at 11:02 AM, Richard Bardwell wrote:
>>>>>> Ralph
>>>>>> I had done a make clean in the 1.2.8 directory if that is what you meant ?
>>>>>> Or do I need to do something else ?
>>>>>> I appreciate your help on this by the way ;-)
>>>>> Hi Richard
>>>>> You can install in a different directory, totally separate from 1.2.8.
>>>>> Create a new work directory [which is not the final installation directory, just work, say /tmp/openmpi/1.4.4/work].
>>>>> Launch the OpenMPI 1.4.4 configure script from this new work directory with the --prefix pointing to your desired installation
>>>>> directory [e.g. /home/richard/openmpi/1.4.4/].
>>>>> I am assuming this is NFS mounted on the nodes [if you have a cluster].
>>>>> [Check all options with 'configure --help'.]
>>>>> Then do make, make install.
>>>>> Finally set your PATH and LD_LIBRARY_PATH to point to the new installation directory,
>>>>> to prevent mixing with the old 1.2.8.
>>>>> I have a number of OpenMPI versions here, compiled with various compilers,
>>>>> and they coexist well this way.
>>>>> I hope this helps,
>>>>> Gus Correa
>>>>>> ----- Original Message -----
>>>>>> From: Ralph Castain
>>>>>> To: Open MPI Users
>>>>>> Sent: Monday, February 13, 2012 3:41 PM
>>>>>> Subject: Re: [OMPI users] MPI orte_init fails on remote nodes
>>>>>> You need to clean out the old attempt - that is a stale file
>>>>>> Sent from my iPad
>>>>>> On Feb 13, 2012, at 7:36 AM, "Richard Bardwell" <richard_at_[hidden]> wrote:
>>>>>>> OK, I installed 1.4.4, rebuilt the exec and guess what ...... I now get some weird errors as below:
>>>>>>> mca: base: component_find: unable to open /usr/local/lib/openmpi/mca_ras_dash_host
>>>>>>> along with a few other files
>>>>>>> even though the .so / .la files are all there !
>>>>>>> ----- Original Message -----
>>>>>>> From: Ralph Castain
>>>>>>> To: Open MPI Users
>>>>>>> Sent: Monday, February 13, 2012 2:59 PM
>>>>>>> Subject: Re: [OMPI users] MPI orte_init fails on remote nodes
>>>>>>> Good heavens - where did you find something that old? Can you use a more recent version?
>>>>>>> Sent from my iPad
>>>>>>>
>>>>>>>> Gentlemen
>>>>>>>> I am struggling to get MPI working when the hostfile contains different nodes.
>>>>>>>> I get the error below. Any ideas ?? I can ssh without password between the two
>>>>>>>> nodes. I am running 1.2.8 MPI on both machines.
>>>>>>>> Any help most appreciated !!!!!
>>>>>>>> MPITEST/v8_mpi_test> mpiexec -n 2 --debug-daemons -hostfile test.hst /home/sharc/MPITEST/v8_mpi_test/mpitest
>>>>>>>> Daemon [0,0,1] checking in as pid 10490 on host 192.0.2.67
>>>>>>>> [linux-z0je:08804] [NO-NAME] ORTE_ERROR_LOG: Not found in file runtime/orte_init_stage1.c at line 182
>>>>>>>> --------------------------------------------------------------------------
>>>>>>>> It looks like orte_init failed for some reason; your parallel process is
>>>>>>>> likely to abort. There are many reasons that a parallel process can
>>>>>>>> fail during orte_init; some of which are due to configuration or
>>>>>>>> environment problems. This failure appears to be an internal failure;
>>>>>>>> here's some additional information (which may only be relevant to an
>>>>>>>> Open MPI developer):
>>>>>>>> orte_rml_base_select failed
>>>>>>>> --> Returned value -13 instead of ORTE_SUCCESS
>>>>>>>> --------------------------------------------------------------------------
>>>>>>>> [linux-z0je:08804] [NO-NAME] ORTE_ERROR_LOG: Not found in file runtime/orte_system_init.c at line 42
>>>>>>>> [linux-z0je:08804] [NO-NAME] ORTE_ERROR_LOG: Not found in file runtime/orte_init.c at line 52
>>>>>>>> Open RTE was unable to initialize properly. The error occured while
>>>>>>>> attempting to orte_init(). Returned value -13 instead of ORTE_SUCCESS.
>>>>>>>> [linux-tmpw:10490] [0,0,1] orted_recv_pls: received message from [0,0,0]
>>>>>>>> [linux-tmpw:10490] [0,0,1] orted_recv_pls: received kill_local_procs
>>>>>>>> [linux-tmpw:10489] [0,0,0] ORTE_ERROR_LOG: Timeout in file base/pls_base_orted_cmds.c at line 275
>>>>>>>> [linux-tmpw:10489] [0,0,0] ORTE_ERROR_LOG: Timeout in file pls_rsh_module.c at line 1158
>>>>>>>> [linux-tmpw:10489] [0,0,0] ORTE_ERROR_LOG: Timeout in file errmgr_hnp.c at line 90
>>>>>>>> [linux-tmpw:10489] ERROR: A daemon on node 192.0.2.68 failed to start as expected.
>>>>>>>> [linux-tmpw:10489] ERROR: There may be more information available from
>>>>>>>> [linux-tmpw:10489] ERROR: the remote shell (see above).
>>>>>>>> [linux-tmpw:10489] ERROR: The daemon exited unexpectedly with status 243.
>>>>>>>> [linux-tmpw:10490] [0,0,1] orted_recv_pls: received message from [0,0,0]
>>>>>>>> [linux-tmpw:10490] [0,0,1] orted_recv_pls: received exit
>>>>>>>> [linux-tmpw:10489] [0,0,0] ORTE_ERROR_LOG: Timeout in file base/pls_base_orted_cmds.c at line 188
>>>>>>>> [linux-tmpw:10489] [0,0,0] ORTE_ERROR_LOG: Timeout in file pls_rsh_module.c at line 1190
>>>>>>>> --------------------------------------------------------------------------
>>>>>>>> mpiexec was unable to cleanly terminate the daemons for this job. Returned value Timeout instead of ORTE_SUCCESS.
>>>>>>>> --------------------------------------------------------------------------
>>>>>>>> _______________________________________________
>>>>>>>> users mailing list
>>>>>>>> users_at_[hidden]
>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>>>> _______________________________________________
>>>>>>> users mailing list
>>>>>>> users_at_[hidden]
>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>>>> _______________________________________________
>>>>>>> users mailing list
>>>>>>> users_at_[hidden]
>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>>> _______________________________________________
>>>>>> users mailing list
>>>>>> users_at_[hidden]
>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>>> _______________________________________________
>>>>>> users mailing list
>>>>>> users_at_[hidden]
>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>> _______________________________________________
>>>>> users mailing list
>>>>> users_at_[hidden]
>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>>
>>>>
>>>> _______________________________________________
>>>> users mailing list
>>>> users_at_[hidden]
>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>
>>>
>>> --
>>> Jeff Squyres
>>> jsquyres_at_[hidden]
>>> For corporate legal information go to:
>>> http://www.cisco.com/web/about/doing_business/legal/cri/
>>>
>>>
>>> _______________________________________________
>>> users mailing list
>>> users_at_[hidden]
>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>
>>
>> _______________________________________________
>> users mailing list
>> users_at_[hidden]
>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>
>
> --
> Jeff Squyres
> jsquyres_at_[hidden]
> For corporate legal information go to:
> http://www.cisco.com/web/about/doing_business/legal/cri/
>
>
> _______________________________________________
> users mailing list
> users_at_[hidden]
> http://www.open-mpi.org/mailman/listinfo.cgi/users
>