On 5/4/2012 1:17 PM, Don Armstrong wrote:
On Fri, 04 May 2012, Rolf vandeVaart wrote:
On Behalf Of Don Armstrong
On Thu, 03 May 2012, Rolf vandeVaart wrote:
2. If that works, then you can also run with a debug switch to
see what connections are being made by MPI.
You can see the connections being made in the attached log:

[archimedes:29820] btl: tcp: attempting to connect() to [[60576,1],2] address
138.23.141.162 on port 2001
Yes, I missed that. So, can we simplify the problem. Can you run
with np=2 and one process on each node?
It hangs in exactly the same spot without completing the initial
sm-based message. [Specifically, the GUID sending and acking appears
to complete on the tcp connection, but the actual traffic is never
sent, and the
ompi_request_wait_completion(&sendreq->req_send.req_base.req_ompi);
never clears).

Also, maybe you can send the ifconfig output from each node. We
sometimes see this type of hanging when a node has two different
interfaces on the same subnet.
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:30:48:7d:82:54 brd ff:ff:ff:ff:ff:ff
    inet 138.23.140.43/23 brd 138.23.141.255 scope global eth0
    inet 172.16.30.79/24 brd 172.16.30.255 scope global eth0:1
    inet6 fe80::230:48ff:fe7d:8254/64 scope link 
       valid_lft forever preferred_lft forever
3: eth1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast state DOWN qlen 1000
    link/ether 00:30:48:7d:82:55 brd ff:ff:ff:ff:ff:ff
    inet6 fd74:56b0:69d6::2101/118 scope global 
       valid_lft forever preferred_lft forever
    inet6 fe80::230:48ff:fe7d:8255/64 scope link 
       valid_lft forever preferred_lft forever
16: tun0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 100
    link/none 
    inet 10.134.0.6/24 brd 10.134.0.255 scope global tun0
17: tun1: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 100
    link/none 
    inet 10.137.0.201/24 brd 10.137.0.255 scope global tun1

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether 00:17:a4:4b:7c:ea brd ff:ff:ff:ff:ff:ff
    inet 172.16.30.110/24 brd 172.16.30.255 scope global eth0:1
    inet 138.23.141.162/23 brd 138.23.141.255 scope global eth0
    inet6 fe80::217:a4ff:fe4b:7cea/64 scope link 
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
    link/ether 00:17:a4:4b:7c:ec brd ff:ff:ff:ff:ff:ff
7: tun0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 100
    link/none 
    inet 10.134.0.26/24 brd 10.134.0.255 scope global tun0

Assuming there are multiple interfaces, can you experiment with the runtime flags outlined here?
http://www.open-mpi.org/faq/?category=tcp#tcp-selection
It's already running with btl_tcp_if_include=eth0 and
oob_tcp_if_include=eth0; the connections are happening only on eth0,
which has the 138.23.141.16 addresses.
Sorry if this is a stupid question but what is eth0:1 (it's under eth0).  Are the 172.16.30.X addresses pingable to each other?


--
Terry D. Dontje | Principal Software Engineer
Developer Tools Engineering | +1.781.442.2631
Oracle - Performance Technologies
95 Network Drive, Burlington, MA 01803
Email terry.dontje@oracle.com