Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: Re: [OMPI users] problem with opal_net_private_ipv4
From: Rolf vandeVaarrt (Rolf.Vandevaart_at_[hidden])
Date: 2010-03-23 13:42:58


Nicolas Niclausse wrote:
> Fernando Lemos ecrivait le 23/03/2010 16:28:
>
>>> I'm trying to run openmpi (1.4.1) on two clusters; on each cluster, several
>>> interfaces are private;
>>>
>>> on cluster1, nodes have 3 interfaces, and only 192.168.159.0/24 is visible
>>> from cluster2.
>>>
>>> chicon-3
>>> eth0 inet addr:192.168.160.76 Bcast:192.168.160.255 Mask:255.255.255.0
>>> eth1 inet addr:192.168.159.76 Bcast:192.168.159.255 Mask:255.255.255.0
>>> myri0 inet addr:192.168.162.76 Bcast:192.168.162.255 Mask:255.255.255.0
>>>
>>> on cluster2, nodes have 3 interfaces, and only 172.24.110.0/17 is visible
>>> from cluster1
>>>
>>> netgdx-8
>>> eth0 inet addr:172.24.190.8 Bcast:172.24.191.255 Mask:255.255.192.0
>>> eth1 inet addr:172.24.110.8 Bcast:172.24.127.255 Mask:255.255.128.0
>>> eth2 inet addr:172.24.240.8 Bcast:172.24.255.255 Mask:255.255.192.0
>>>
>>> so i'm using this to declare all the other networks as private:
>>>
>>> mpirun -machinefile ~/gridnodes --mca opal_net_private_ipv4
>>> "192.168.162.0/24\;192.168.160.0/24\;172.24.192.0/18\;172.24.128.0/18"
>>> ./alltoall
>>>
>>> but this doesn't work:
>>>
>> Have you tried -mca btl_tcp_if_include/exclude?
>>
>
> I can't do that because the "public" interface is not always eth1 as in
> this example (i have several other clusters with different network
> configurations in my setup)
>
>
>>> Why openmpi tries to connect different private networks, given that
>>> "public" networks exists ? is it a bug or am i missing something ?
>>>
>> >From what I've seen, I believe OpenMPI tries to find the fastest route
>> to the nodes. In some cases it's trivial to sort that out, in other
>> cases you might need to give it some hints.
>>
>
> yes, so i thought that "opal_net_private_ipv4" was the right thing for me;
> but it doesn't work without the patch.
>
>
It seems to me that you are entering a piece of the code where the code
thinks at least one of the interfaces is private. And when comparing a
public and private, it gives a weighting of
CQ_PRIVATE_DIFFERENT_NETWORK. I am not sure why, but that is the
weighting it gives. You can take a look at this FAQ
http://www.open-mpi.org/faq/?category=tcp#tcp-routability-1.3 which has
links to the paper that explains how all this logic works.

It seems that what you are doing makes sense. You are trying to define
which networks are private so that in the end you
expect the two other networks to end up being public, and therefore get
the highest weight for a connection.

I realize this does not help much, but maybe the paper will help out.

Rolf