Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: Re: [OMPI users] problem using new OMPI1.4.1 vie SGE
From: Addepalli, Srirangam V (srirangam.v.addepalli_at_[hidden])
Date: 2010-04-26 11:57:04


Hello Matthew,

what are the build flags you used for building openmpi on RANGER. This info should be in the config.log file.

eg: head -10 config.log | grep './configure'
 $ ./configure --with-openib=/usr/ -with-ft=cr --with-blcr=/usr --with-blcr-libdir=/usr/lib64/ --prefix=/lustre/work/saddepal/openmpi-1.4-BM CC=icc FC=ifort

Rangam
________________________________________
From: users-bounces_at_[hidden] [users-bounces_at_[hidden]] On Behalf Of Matthew MacManes [macmanes_at_[hidden]]
Sent: Monday, April 26, 2010 10:37 AM
To: Open MPI Users
Subject: Re: [OMPI users] problem using new OMPI1.4.1 vie SGE

Hi Rangam,

Thanks.. Just tried. Still the 'no route to host' issue..

Matt
_________________________________
Matthew MacManes
PhD Candidate
University of California- Berkeley
Museum of Vertebrate Zoology
Phone: 510-495-5833
Lab Website: http://ib.berkeley.edu/labs/lacey
Personal Website: http://macmanes.com/

On Mon, Apr 26, 2010 at 08:14, Addepalli, Srirangam V <srirangam.v.addepalli_at_[hidden]<mailto:srirangam.v.addepalli_at_[hidden]>> wrote:
Hello Matthew,

Not sure it it helps but I am guessing that

module load openmpi (in bash script) is updating variables to RANGER openmpi installation. Try removing this line in your script file and resubmit your job, also
#$ -V in the submission script exports env variables from your current terminal. Check if you have the correct setting before your submit your job.

Rangam

______________________________ __________
From: users-bounces_at_[hidden]<mailto:users-bounces_at_[hidden]> [users-bounces_at_[hidden]<mailto:users-bounces_at_[hidden]>] On Behalf Of Matthew MacManes [macmanes_at_[hidden]<mailto:macmanes_at_[hidden]>]
Sent: Monday, April 26, 2010 10:08 AM
To: Open Users
Subject: [OMPI users] problem using new OMPI1.4.1 vie SGE

I am using SGE to submit jobs to one of the TeraGrid sites, specifically TACC-RANGER. The problem, is, that I am using a program that requires OpenMPI version 1.4.1, and the latest install on RANGER is 1.3.1. I was told that I could install OpenMPI in my home directory, and run jobs using my newer version.. However, I am having problems doing this, getting the error message seen below.

Its seems that the compute nodes are not accessing all the sufficient libraries for the newer version of OpenMPI.

Can anybody tell me what I can do to get the jobs running using the newer version of OpenMPI. Thanks!

TACC: Setting memory limits for job 1349843 to 3984588 KB
TACC: Dumping job script:
------------------------------ --------------------------------------------------
#!/bin/bash
export TMPDIR=$SCRATCH/abyss_tmp/
LD_LIBRARY_PATH=/work/01301/mmacmane
LD_LIBRARY_PATH=/work/01301/mmacmane/bin
LD_LIBRARY_PATH=/work/01301/mmacmane/include
LD_LIBRARY_PATH=/work/01301/mmacmane/etc
LD_LIBRARY_PATH=/work/01301/mmacmane/lib
LD_LIBRARY_PATH=/work/01301/mmacmane/openmpi-1.4.1
cd /work/01301/mmacmane/Ray-0.0.6
module load openmpi
#$ -N testing_MRNA2
#$ -j y
#$ -o /work/01301/mmacmane/Ray-0.0.6/testing_MRNA2
#$ -pe 8way 128
#$ -q normal
#$ -l h_rt=2:00:00
#$ -M macmanes_at_[hidden]<mailto:macmanes_at_[hidden]><mailto:macmanes_at_[hidden]<mailto:macmanes_at_[hidden]>>
#$ -m be
#$ -cwd
#$ -V
/work/01301/mmacmane/bin/mpirun Ray /work/01301/mmacmane/Ray-0.0.6/Ray_snp.txt--------------------------------------------------------------------------------
TACC: Done.
   Module mvapich superceded

Ray Copyright (C) 2010 Sébastien Boisvert, Jacques Corbeil, François Laviolette
http://denovoassembler.sf.net/
This program comes with ABSOLUTELY NO WARRANTY.
This is free software, and you are welcome to redistribute it
under certain conditions; see "gpl-3.0.txt" for details.
[i180-212.ranger.tacc.utexas.edu<http://i180-212.ranger.tacc.utexas.edu>][[29053,1],114][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] connect() to 192.168.12.104 failed: No route to host (113)
[i180-212.ranger.tacc.utexas.edu<http://i180-212.ranger.tacc.utexas.edu>][[29053,1],119][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] connect() to 192.168.12.104 failed: No route to host (113)
[i180-212.ranger.tacc.utexas.edu<http://i180-212.ranger.tacc.utexas.edu>][[29053,1],123][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] connect() to 192.168.12.104 failed: No route to host (113)
[i161-311.ranger.tacc.utexas.edu<http://i161-311.ranger.tacc.utexas.edu>][[29053,1],17][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] connect() to 192.168.5.170 failed: No route to host (113)
[i128-412.ranger.tacc.utexas.edu<http://i128-412.ranger.tacc.utexas.edu>][[29053,1],42][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] [i128-412.ranger.tacc.utexas.edu<http://i128-412.ranger.tacc.utexas.edu>][[29053,1],44][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] connect() to 192.168.4.244 failed: No route to host (113)
[i120-302.ranger.tacc.utexas.edu<http://i120-302.ranger.tacc.utexas.edu>][[29053,1],1][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] connect() to 192.168.13.99 failed: No route to host (113)
[i120-302.ranger.tacc.utexas.edu<http://i120-302.ranger.tacc.utexas.edu>][[29053,1],13][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] connect() to 192.168.13.99 failed: No route to host (113)
[i120-302.ranger.tacc.utexas.edu<http://i120-302.ranger.tacc.utexas.edu>][[29053,1],9][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] connect() to 192.168.13.99 failed: No route to host (113)
[i156-212.ranger.tacc.utexas.edu<http://i156-212.ranger.tacc.utexas.edu>][[29053,1],104][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] connect() to 192.168.16.232 failed: No route to host (113)
[i156-212.ranger.tacc.utexas.edu<http://i156-212.ranger.tacc.utexas.edu>][[29053,1],106][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] connect() to 192.168.16.232 failed: No route to host (113)
[i156-212.ranger.tacc.utexas.edu<http://i156-212.ranger.tacc.utexas.edu>][[29053,1],102][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] connect() to 192.168.16.232 failed: No route to host (113)
[i128-412.ranger.tacc.utexas.edu<http://i128-412.ranger.tacc.utexas.edu>][[29053,1],45][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] connect() to 192.168.4.244 failed: No route to host (113)
connect() to 192.168.4.244 failed: No route to host (113)
[i170-204.ranger.tacc.utexas.edu<http://i170-204.ranger.tacc.utexas.edu>][[29053,1],83][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] connect() to 192.168.2.196 failed: No route to host (113)
[i170-204.ranger.tacc.utexas.edu<http://i170-204.ranger.tacc.utexas.edu>][[29053,1],84][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] connect() to 192.168.2.196 failed: No route to host (113)
[i170-204.ranger.tacc.utexas.edu<http://i170-204.ranger.tacc.utexas.edu>][[29053,1],92][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] connect() to 192.168.2.196 failed: No route to host (113)
[i105-104.ranger.tacc.utexas.edu<http://i105-104.ranger.tacc.utexas.edu>][[29053,1],66][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] connect() to 192.168.15.0 failed: No route to host (113)
[i105-104.ranger.tacc.utexas.edu<http://i105-104.ranger.tacc.utexas.edu>][[29053,1],70][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] connect() to 192.168.15.0 failed: No route to host (113)
[i116-312.ranger.tacc.utexas.edu<http://i116-312.ranger.tacc.utexas.edu>][[29053,1],52][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] connect() to 192.168.7.64 failed: No route to host (113)
[i116-312.ranger.tacc.utexas.edu<http://i116-312.ranger.tacc.utexas.edu>][[29053,1],60][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] connect() to 192.168.7.64 failed: No route to host (113)
[i116-312.ranger.tacc.utexas.edu<http://i116-312.ranger.tacc.utexas.edu>][[29053,1],58][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] connect() to 192.168.7.64 failed: No route to host (113)
[i105-104.ranger.tacc.utexas.edu<http://i105-104.ranger.tacc.utexas.edu>][[29053,1],72][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] connect() to 192.168.15.0 failed: No route to host (113)
[i161-311.ranger.tacc.utexas.edu<http://i161-311.ranger.tacc.utexas.edu>][[29053,1],23][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] connect() to 192.168.5.170 failed: No route to host (113)
[i161-311.ranger.tacc.utexas.edu<http://i161-311.ranger.tacc.utexas.edu>][[29053,1],29][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] connect() to 192.168.5.170 failed: No route to host (113)
[i161-311.ranger.tacc.utexas.edu<http://i161-311.ranger.tacc.utexas.edu>][[29053,1],31][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] connect() to 192.168.5.170 failed: No route to host (113)
[i128-412.ranger.tacc.utexas.edu<http://i128-412.ranger.tacc.utexas.edu>][[29053,1],35][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] connect() to 192.168.4.244 failed: No route to host (113)
[i128-412.ranger.tacc.utexas.edu<http://i128-412.ranger.tacc.utexas.edu>][[29053,1],43][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] connect() to 192.168.4.244 failed: No route to host (113)
[i120-302.ranger.tacc.utexas.edu<http://i120-302.ranger.tacc.utexas.edu>][[29053,1],0][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] connect() to 192.168.13.99 failed: No route to host (113)
[i120-302.ranger.tacc.utexas.edu<http://i120-302.ranger.tacc.utexas.edu>][[29053,1],6][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] connect() to 192.168.13.99 failed: No route to host (113)
[i120-302.ranger.tacc.utexas.edu<http://i120-302.ranger.tacc.utexas.edu>][[29053,1],14][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] connect() to 192.168.13.99 failed: No route to host (113)
[i105-104.ranger.tacc.utexas.edu<http://i105-104.ranger.tacc.utexas.edu>][[29053,1],73][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] [i105-104.ranger.tacc.utexas.edu<http://i105-104.ranger.tacc.utexas.edu>][[29053,1],77][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] connect() to 192.168.15.0 failed: No route to host (113)
connect() to 192.168.15.0 failed: No route to host (113)
[i105-104.ranger.tacc.utexas.edu<http://i105-104.ranger.tacc.utexas.edu>][[29053,1],75][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] connect() to 192.168.15.0 failed: No route to host (113)
[i156-212.ranger.tacc.utexas.edu<http://i156-212.ranger.tacc.utexas.edu>][[29053,1],99][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] [i156-212.ranger.tacc.utexas.edu<http://i156-212.ranger.tacc.utexas.edu>][[29053,1],109][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] connect() to 192.168.16.232 failed: No route to host (113)
[i156-212.ranger.tacc.utexas.edu<http://i156-212.ranger.tacc.utexas.edu>][[29053,1],103][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] connect() to 192.168.16.232 failed: No route to host (113)
connect() to 192.168.16.232 failed: No route to host (113)
[i116-312.ranger.tacc.utexas.edu<http://i116-312.ranger.tacc.utexas.edu>][[29053,1],51][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] connect() to 192.168.7.64 failed: No route to host (113)
[i116-312.ranger.tacc.utexas.edu<http://i116-312.ranger.tacc.utexas.edu>][[29053,1],55][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] connect() to 192.168.7.64 failed: No route to host (113)
[i116-312.ranger.tacc.utexas.edu<http://i116-312.ranger.tacc.utexas.edu>][[29053,1],57][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] connect() to 192.168.7.64 failed: No route to host (113)
[i180-212.ranger.tacc.utexas.edu<http://i180-212.ranger.tacc.utexas.edu>][[29053,1],113][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] connect() to 192.168.12.104 failed: No route to host (113)
[i180-212.ranger.tacc.utexas.edu<http://i180-212.ranger.tacc.utexas.edu>][[29053,1],116][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] connect() to 192.168.12.104 failed: No route to host (113)
[i180-212.ranger.tacc.utexas.edu<http://i180-212.ranger.tacc.utexas.edu>][[29053,1],115][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] connect() to 192.168.12.104 failed: No route to host (113)
[i161-311.ranger.tacc.utexas.edu<http://i161-311.ranger.tacc.utexas.edu>][[29053,1],19][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] connect() to 192.168.5.170 failed: No route to host (113)
[i161-311.ranger.tacc.utexas.edu<http://i161-311.ranger.tacc.utexas.edu>][[29053,1],21][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] connect() to 192.168.5.170 failed: No route to host (113)
[i161-311.ranger.tacc.utexas.edu<http://i161-311.ranger.tacc.utexas.edu>][[29053,1],27][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] connect() to 192.168.5.170 failed: No route to host (113)
[i128-412.ranger.tacc.utexas.edu<http://i128-412.ranger.tacc.utexas.edu>][[29053,1],37][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] connect() to 192.168.4.244 failed: No route to host (113)
[i128-412.ranger.tacc.utexas.edu<http://i128-412.ranger.tacc.utexas.edu>][[29053,1],47][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] connect() to 192.168.4.244 failed: No route to host (113)
[i128-412.ranger.tacc.utexas.edu<http://i128-412.ranger.tacc.utexas.edu>][[29053,1],33][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] connect() to 192.168.4.244 failed: No route to host (113)
[i120-302.ranger.tacc.utexas.edu<http://i120-302.ranger.tacc.utexas.edu>][[29053,1],8][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] connect() to 192.168.13.99 failed: No route to host (113)
[i120-302.ranger.tacc.utexas.edu<http://i120-302.ranger.tacc.utexas.edu>][[29053,1],10][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] connect() to 192.168.13.99 failed: No route to host (113)
[i120-302.ranger.tacc.utexas.edu<http://i120-302.ranger.tacc.utexas.edu>][[29053,1],4][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] connect() to 192.168.13.99 failed: No route to host (113)
[i156-212.ranger.tacc.utexas.edu<http://i156-212.ranger.tacc.utexas.edu>][[29053,1],97][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] connect() to 192.168.16.232 failed: No route to host (113)
[i156-212.ranger.tacc.utexas.edu<http://i156-212.ranger.tacc.utexas.edu>][[29053,1],101][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] connect() to 192.168.16.232 failed: No route to host (113)
[i156-212.ranger.tacc.utexas.edu<http://i156-212.ranger.tacc.utexas.edu>][[29053,1],107][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] connect() to 192.168.16.232 failed: No route to host (113)
[i170-204.ranger.tacc.utexas.edu<http://i170-204.ranger.tacc.utexas.edu>][[29053,1],82][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] connect() to 192.168.2.196 failed: No route to host (113)
[i170-204.ranger.tacc.utexas.edu<http://i170-204.ranger.tacc.utexas.edu>][[29053,1],85][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] connect() to 192.168.2.196 failed: No route to host (113)
[i170-204.ranger.tacc.utexas.edu<http://i170-204.ranger.tacc.utexas.edu>][[29053,1],90][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] connect() to 192.168.2.196 failed: No route to host (113)
[i105-104.ranger.tacc.utexas.edu<http://i105-104.ranger.tacc.utexas.edu>][[29053,1],79][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] connect() to 192.168.15.0 failed: No route to host (113)
[i105-104.ranger.tacc.utexas.edu<http://i105-104.ranger.tacc.utexas.edu>][[29053,1],65][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] connect() to 192.168.15.0 failed: No route to host (113)
[i105-104.ranger.tacc.utexas.edu<http://i105-104.ranger.tacc.utexas.edu>][[29053,1],67][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] connect() to 192.168.15.0 failed: No route to host (113)
[i116-312.ranger.tacc.utexas.edu<http://i116-312.ranger.tacc.utexas.edu>][[29053,1],61][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] connect() to 192.168.7.64 failed: No route to host (113)
[i116-312.ranger.tacc.utexas.edu<http://i116-312.ranger.tacc.utexas.edu>][[29053,1],53][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] connect() to 192.168.7.64 failed: No route to host (113)
[i116-312.ranger.tacc.utexas.edu<http://i116-312.ranger.tacc.utexas.edu>][[29053,1],59][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] connect() to 192.168.7.64 failed: No route to host (113)
[i180-212.ranger.tacc.utexas.edu<http://i180-212.ranger.tacc.utexas.edu>][[29053,1],127][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] connect() to 192.168.12.104 failed: No route to host (113)
[i180-212.ranger.tacc.utexas.edu<http://i180-212.ranger.tacc.utexas.edu>][[29053,1],121][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] connect() to 192.168.12.104 failed: No route to host (113)
[i180-212.ranger.tacc.utexas.edu<http://i180-212.ranger.tacc.utexas.edu>][[29053,1],124][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] connect() to 192.168.12.104 failed: No route to host (113)
[i161-311.ranger.tacc.utexas.edu<http://i161-311.ranger.tacc.utexas.edu>][[29053,1],18][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] connect() to 192.168.5.170 failed: No route to host (113)
[i161-311.ranger.tacc.utexas.edu<http://i161-311.ranger.tacc.utexas.edu>][[29053,1],25][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] connect() to 192.168.5.170 failed: No route to host (113)
[i161-311.ranger.tacc.utexas.edu<http://i161-311.ranger.tacc.utexas.edu>][[29053,1],28][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] connect() to 192.168.5.170 failed: No route to host (113)
[i128-412.ranger.tacc.utexas.edu<http://i128-412.ranger.tacc.utexas.edu>][[29053,1],39][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] connect() to 192.168.4.244 failed: No route to host (113)
[i128-412.ranger.tacc.utexas.edu<http://i128-412.ranger.tacc.utexas.edu>][[29053,1],34][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] connect() to 192.168.4.244 failed: No route to host (113)
[i128-412.ranger.tacc.utexas.edu<http://i128-412.ranger.tacc.utexas.edu>][[29053,1],38][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] connect() to 192.168.4.244 failed: No route to host (113)
[i120-302.ranger.tacc.utexas.edu<http://i120-302.ranger.tacc.utexas.edu>][[29053,1],3][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] connect() to 192.168.13.99 failed: No route to host (113)
[i120-302.ranger.tacc.utexas.edu<http://i120-302.ranger.tacc.utexas.edu>][[29053,1],2][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] connect() to 192.168.13.99 failed: No route to host (113)
[i120-302.ranger.tacc.utexas.edu<http://i120-302.ranger.tacc.utexas.edu>][[29053,1],12][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] connect() to 192.168.13.99 failed: No route to host (113)
[i156-212.ranger.tacc.utexas.edu<http://i156-212.ranger.tacc.utexas.edu>][[29053,1],105][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] connect() to 192.168.16.232 failed: No route to host (113)
[i156-212.ranger.tacc.utexas.edu<http://i156-212.ranger.tacc.utexas.edu>][[29053,1],108][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] connect() to 192.168.16.232 failed: No route to host (113)
[i156-212.ranger.tacc.utexas.edu<http://i156-212.ranger.tacc.utexas.edu>][[29053,1],111][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] connect() to 192.168.16.232 failed: No route to host (113)
[i170-204.ranger.tacc.utexas.edu<http://i170-204.ranger.tacc.utexas.edu>][[29053,1],91][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] connect() to 192.168.2.196 failed: No route to host (113)
[i170-204.ranger.tacc.utexas.edu<http://i170-204.ranger.tacc.utexas.edu>][[29053,1],80][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] connect() to 192.168.2.196 failed: No route to host (113)
[i170-204.ranger.tacc.utexas.edu<http://i170-204.ranger.tacc.utexas.edu>][[29053,1],87][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] connect() to 192.168.2.196 failed: No route to host (113)
[i105-104.ranger.tacc.utexas.edu<http://i105-104.ranger.tacc.utexas.edu>][[29053,1],69][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] connect() to 192.168.15.0 failed: No route to host (113)
[i105-104.ranger.tacc.utexas.edu<http://i105-104.ranger.tacc.utexas.edu>][[29053,1],68][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] connect() to 192.168.15.0 failed: No route to host (113)
[i105-104.ranger.tacc.utexas.edu<http://i105-104.ranger.tacc.utexas.edu>][[29053,1],71][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] connect() to 192.168.15.0 failed: No route to host (113)
[i116-312.ranger.tacc.utexas.edu<http://i116-312.ranger.tacc.utexas.edu>][[29053,1],63][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] [i116-312.ranger.tacc.utexas.edu<http://i116-312.ranger.tacc.utexas.edu>][[29053,1],48][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] connect() to 192.168.7.64 failed: No route to host (113)
[i116-312.ranger.tacc.utexas.edu<http://i116-312.ranger.tacc.utexas.edu>][[29053,1],49][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] connect() to 192.168.7.64 failed: No route to host (113)
connect() to 192.168.7.64 failed: No route to host (113)
[i161-311.ranger.tacc.utexas.edu<http://i161-311.ranger.tacc.utexas.edu>][[29053,1],16][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] connect() to 192.168.5.170 failed: No route to host (113)
[i161-311.ranger.tacc.utexas.edu<http://i161-311.ranger.tacc.utexas.edu>][[29053,1],24][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] connect() to 192.168.5.170 failed: No route to host (113)
[i161-311.ranger.tacc.utexas.edu<http://i161-311.ranger.tacc.utexas.edu>][[29053,1],26][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] connect() to 192.168.5.170 failed: No route to host (113)
[i180-212.ranger.tacc.utexas.edu<http://i180-212.ranger.tacc.utexas.edu>][[29053,1],125][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] connect() to 192.168.12.104 failed: No route to host (113)
[i180-212.ranger.tacc.utexas.edu<http://i180-212.ranger.tacc.utexas.edu>][[29053,1],122][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] connect() to 192.168.12.104 failed: No route to host (113)
[i180-212.ranger.tacc.utexas.edu<http://i180-212.ranger.tacc.utexas.edu>][[29053,1],126][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] connect() to 192.168.12.104 failed: No route to host (113)
[i128-412.ranger.tacc.utexas.edu<http://i128-412.ranger.tacc.utexas.edu>][[29053,1],40][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] connect() to 192.168.4.244 failed: No route to host (113)
[i128-412.ranger.tacc.utexas.edu<http://i128-412.ranger.tacc.utexas.edu>][[29053,1],41][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] connect() to 192.168.4.244 failed: No route to host (113)
[i128-412.ranger.tacc.utexas.edu<http://i128-412.ranger.tacc.utexas.edu>][[29053,1],46][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] connect() to 192.168.4.244 failed: No route to host (113)
[i120-302.ranger.tacc.utexas.edu<http://i120-302.ranger.tacc.utexas.edu>][[29053,1],15][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] connect() to 192.168.13.99 failed: No route to host (113)
[i120-302.ranger.tacc.utexas.edu<http://i120-302.ranger.tacc.utexas.edu>][[29053,1],7][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] connect() to 192.168.13.99 failed: No route to host (113)
[i120-302.ranger.tacc.utexas.edu<http://i120-302.ranger.tacc.utexas.edu>][[29053,1],11][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] connect() to 192.168.13.99 failed: No route to host (113)
[i156-212.ranger.tacc.utexas.edu<http://i156-212.ranger.tacc.utexas.edu>][[29053,1],100][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] connect() to 192.168.16.232 failed: No route to host (113)
[i156-212.ranger.tacc.utexas.edu<http://i156-212.ranger.tacc.utexas.edu>][[29053,1],110][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] connect() to 192.168.16.232 failed: No route to host (113)
[i156-212.ranger.tacc.utexas.edu<http://i156-212.ranger.tacc.utexas.edu>][[29053,1],96][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] connect() to 192.168.16.232 failed: No route to host (113)
[i170-204.ranger.tacc.utexas.edu<http://i170-204.ranger.tacc.utexas.edu>][[29053,1],88][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] connect() to 192.168.2.196 failed: No route to host (113)
[i170-204.ranger.tacc.utexas.edu<http://i170-204.ranger.tacc.utexas.edu>][[29053,1],89][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] connect() to 192.168.2.196 failed: No route to host (113)
[i170-204.ranger.tacc.utexas.edu<http://i170-204.ranger.tacc.utexas.edu>][[29053,1],94][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] connect() to 192.168.2.196 failed: No route to host (113)
[i105-104.ranger.tacc.utexas.edu<http://i105-104.ranger.tacc.utexas.edu>][[29053,1],76][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] connect() to 192.168.15.0 failed: No route to host (113)
[i105-104.ranger.tacc.utexas.edu<http://i105-104.ranger.tacc.utexas.edu>][[29053,1],64][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] connect() to 192.168.15.0 failed: No route to host (113)
[i105-104.ranger.tacc.utexas.edu<http://i105-104.ranger.tacc.utexas.edu>][[29053,1],78][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] connect() to 192.168.15.0 failed: No route to host (113)
[i116-312.ranger.tacc.utexas.edu<http://i116-312.ranger.tacc.utexas.edu>][[29053,1],62][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] connect() to 192.168.7.64 failed: No route to host (113)
[i116-312.ranger.tacc.utexas.edu<http://i116-312.ranger.tacc.utexas.edu>][[29053,1],50][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] [i116-312.ranger.tacc.utexas.edu<http://i116-312.ranger.tacc.utexas.edu>][[29053,1],56][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] connect() to 192.168.7.64 failed: No route to host (113)
connect() to 192.168.7.64 failed: No route to host (113)
[i161-311.ranger.tacc.utexas.edu<http://i161-311.ranger.tacc.utexas.edu>][[29053,1],20][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] connect() to 192.168.5.170 failed: No route to host (113)
[i161-311.ranger.tacc.utexas.edu<http://i161-311.ranger.tacc.utexas.edu>][[29053,1],22][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] [i161-311.ranger.tacc.utexas.edu<http://i161-311.ranger.tacc.utexas.edu>][[29053,1],30][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] connect() to 192.168.5.170 failed: No route to host (113)
connect() to 192.168.5.170 failed: No route to host (113)
[i180-212.ranger.tacc.utexas.edu<http://i180-212.ranger.tacc.utexas.edu>][[29053,1],118][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] connect() to 192.168.12.104 failed: No route to host (113)
[i180-212.ranger.tacc.utexas.edu<http://i180-212.ranger.tacc.utexas.edu>][[29053,1],112][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] connect() to 192.168.12.104 failed: No route to host (113)
[i180-212.ranger.tacc.utexas.edu<http://i180-212.ranger.tacc.utexas.edu>][[29053,1],117][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] connect() to 192.168.12.104 failed: No route to host (113)
[i128-412.ranger.tacc.utexas.edu<http://i128-412.ranger.tacc.utexas.edu>][[29053,1],32][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] connect() to 192.168.4.244 failed: No route to host (113)
[i128-412.ranger.tacc.utexas.edu<http://i128-412.ranger.tacc.utexas.edu>][[29053,1],36][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] connect() to 192.168.4.244 failed: No route to host (113)
[i120-302.ranger.tacc.utexas.edu<http://i120-302.ranger.tacc.utexas.edu>][[29053,1],5][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] connect() to 192.168.13.99 failed: No route to host (113)
[i156-212.ranger.tacc.utexas.edu<http://i156-212.ranger.tacc.utexas.edu>][[29053,1],98][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] connect() to 192.168.16.232 failed: No route to host (113)
[i170-204.ranger.tacc.utexas.edu<http://i170-204.ranger.tacc.utexas.edu>][[29053,1],86][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] connect() to 192.168.2.196 failed: No route to host (113)
[i170-204.ranger.tacc.utexas.edu<http://i170-204.ranger.tacc.utexas.edu>][[29053,1],95][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] connect() to 192.168.2.196 failed: No route to host (113)
[i170-204.ranger.tacc.utexas.edu<http://i170-204.ranger.tacc.utexas.edu>][[29053,1],93][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] connect() to 192.168.2.196 failed: No route to host (113)
[i105-104.ranger.tacc.utexas.edu<http://i105-104.ranger.tacc.utexas.edu>][[29053,1],74][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] connect() to 192.168.15.0 failed: No route to host (113)
[i116-312.ranger.tacc.utexas.edu<http://i116-312.ranger.tacc.utexas.edu>][[29053,1],54][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] connect() to 192.168.7.64 failed: No route to host (113)
[i180-212.ranger.tacc.utexas.edu<http://i180-212.ranger.tacc.utexas.edu>][[29053,1],120][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] connect() to 192.168.12.104 failed: Connection timed out (110)
[i170-204.ranger.tacc.utexas.edu<http://i170-204.ranger.tacc.utexas.edu>][[29053,1],81][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] connect() to 192.168.2.196 failed: Connection timed out (110)
--------------------------------------------------------------------------
A daemon (pid 24537) died unexpectedly with status 137 while attempting
to launch so we are aborting.

There may be more information reported by the environment (see above).

This may be because the daemon was unable to find all the needed shared
libraries on the remote node. You may set your LD_LIBRARY_PATH to have the
location of the shared libraries on the remote nodes and this will
automatically be forwarded to the remote nodes.
------------------------------ --------------------------------------------
--------------------------------------------------------------------------
mpirun noticed that the job aborted, but has no info as to the process
that caused that situation.
--------------------------------------------------------------------------
[i120-302.ranger.tacc.utexas.edu:24530<http://i120-302.ranger.tacc.utexas.edu:24530>] [[29053,0],0]-[[29053,0],4] mca_oob_tcp_msg_recv: readv failed: Connection reset by peer (104)
--------------------------------------------------------------------------
mpirun was unable to cleanly terminate the daemons on the nodes shown
below. Additional manual cleanup may be required - please refer to
the "orte-clean" tool for assistance.
--------------------------------------------------------------------------
   i128-412.ranger.tacc.utexas.edu<http://i128-412.ranger.tacc.utexas.edu>
   i105-104.ranger.tacc.utexas.edu<http://i105-104.ranger.tacc.utexas.edu>
   i170-204.ranger.tacc.utexas.edu<http://i170-204.ranger.tacc.utexas.edu>
[i161-311.ranger.tacc.utexas.edu:28177<http://i161-311.ranger.tacc.utexas.edu:28177>] [[29053,0],1] routed:binomial: Connection to lifeline [[29053,0],0] lost
[i156-212.ranger.tacc.utexas.edu:16331<http://i156-212.ranger.tacc.utexas.edu:16331>] [[29053,0],6] routed:binomial: Connection to lifeline [[29053,0],0] lost
[i180-212.ranger.tacc.utexas.edu:09688<http://i180-212.ranger.tacc.utexas.edu:09688>] [[29053,0],7] routed:binomial: Connection to lifeline [[29053,0],0] lost
TACC: Cleaning up after job: 1349843
TACC: Done.
_________________________________
Matthew MacManes
PhD Candidate
University of California- Berkeley
Museum of Vertebrate Zoology
Phone: 510-495-5833
Lab Website: http://ib.berkeley.edu/labs/lacey
Personal Website: http://macmanes.com/

_______________________________________________
users mailing list
users_at_[hidden]<mailto:users_at_[hidden]>
http://www.open-mpi.org/mailman/listinfo.cgi/users