Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: Re: [OMPI users] mpiexec seems to be resolving names on server insteadof each node
From: Rolf vandeVaarrt (Rolf.Vandevaart_at_[hidden])
Date: 2009-05-06 18:14:19


It is worth mentioning that there is a FAQ that describes the algorithm.
http://www.open-mpi.org/faq/?category=tcp#tcp-routability-1.3

Jeff Squyres wrote:
> Sorry for the delay in replying; I kept starting to look into this and
> then getting distracted by shiny objects. :-(
>
> OMPI v1.3 actually has a fairly sophisticated TCP address/network
> matching algorithm. The hostname resolution shouldn't really be the
> issue; OMPI directly queries the kernel IP interfaces on each node
> where it runs and publishes that to all other MPI processes. Then the
> TCP network matching algorithm is used to select the "best" pairs to
> connect to.
>
> Per your diagram:
>
>> 192.168.1.1 192.168.1.2
>> hubert ------------------------ fry
>> | \ / | 192.168.4.1
>> | \ / |
>> | \ / |
>> | \ / |
>> | / \ |
>> | / \ |
>> | / \ |
>> | / \ | 192.168.4.2
>> hermes ----------------------- leelas
>
> I assume that the netmasks are all class C, right?
>
> If so, I might ask you to run some diagnostic OMPI builds so that we
> can see what the matching algorithm is doing on your machine...
>
>
>
> On Apr 17, 2009, at 8:45 PM, Micha Feigin wrote:
>
>> I am having problems running openmpi 1.3 on my claster and I was
>> wondering if
>> anyone else is seeing this problem and/or can give hints on how to
>> solve it
>>
>> As far as I understand the error, mpiexec resolves host names on the
>> master node
>> it is run on instead of an each host seperately. This works in an
>> environment where
>> each hostname resolves to the same address on each host (cluster
>> connected via a
>> switch) but fails where it resolves to different addresses (ring/star
>> setups for
>> example where each computer is connected directly to all/some of the
>> others)
>>
>> I'm not 100% sure that this is the problem as I'm seeing success on a
>> single
>> case where this should probably fail but it is my best bet from the
>> error message.
>>
>> version 1.2.8 worked fine for the same simple program (a simple
>> hellow world that
>> just comunicated the computer name for each process)
>>
>> An example output:
>>
>> mpiexec is run on the master node hubert and is set to run the
>> processes on two nodes
>> fry and leela. As is understood from the error messages leela tries
>> to connect to
>> fry on address 192.168.1.2 which is it's address on hubert but not
>> leela (where it
>> is 192.168.4.1)
>>
>> This is a four node claster all interconnected
>>
>> 192.168.1.1 192.168.1.2
>> hubert ------------------------ fry
>> | \ / | 192.168.4.1
>> | \ / |
>> | \ / |
>> | \ / |
>> | / \ |
>> | / \ |
>> | / \ |
>> | / \ | 192.168.4.2
>> hermes ----------------------- leelas
>>
>> =================================================================
>> mpiexec -np 8 -H fry,leela test_mpi
>> Hello MPI from the server process of 8 on fry!
>> [[36620,1],1][../../../../../../ompi/mca/btl/tcp/btl_tcp_endpoint.c:589:mca_btl_tcp_endpoint_start_connect]
>> from leela to: fry Unable to connect to the peer 192.168.1.2 on port
>> 154: Network is unreachable
>>
>> [[36620,1],3][../../../../../../ompi/mca/btl/tcp/btl_tcp_endpoint.c:589:mca_btl_tcp_endpoint_start_connect]
>> from leela to: fry Unable to connect to the peer 192.168.1.2 on port
>> 154: Network is unreachable
>>
>> [[36620,1],7][../../../../../../ompi/mca/btl/tcp/btl_tcp_endpoint.c:589:mca_btl_tcp_endpoint_start_connect]
>> from leela to: fry Unable to connect to the peer 192.168.1.2 on port
>> 154: Network is unreachable
>>
>> [leela:4436] *** An error occurred in MPI_Send
>> [leela:4436] *** on communicator MPI_COMM_WORLD
>> [leela:4436] *** MPI_ERR_INTERN: internal error
>> [leela:4436] *** MPI_ERRORS_ARE_FATAL (your MPI job will now abort)
>> [[36620,1],5][../../../../../../ompi/mca/btl/tcp/btl_tcp_endpoint.c:589:mca_btl_tcp_endpoint_start_connect]
>> from leela to: fry Unable to connect to the peer 192.168.1.2 on port
>> 154: Network is unreachable
>>
>> --------------------------------------------------------------------------
>>
>> mpiexec has exited due to process rank 1 with PID 4433 on
>> node leela exiting without calling "finalize". This may
>> have caused other processes in the application to be
>> terminated by signals sent by mpiexec (as reported here).
>> --------------------------------------------------------------------------
>>
>> [hubert:11312] 3 more processes have sent help message
>> help-mpi-errors.txt / mpi_errors_are_fatal
>> [hubert:11312] Set MCA parameter "orte_base_help_aggregate" to 0 to
>> see all help / error messages
>> =================================================================
>>
>> This seems to be a directional issue as running the program -H
>> fry,leela failes
>> where -H leela,fry works, same behaviour for all senarious except
>> those that include
>> the master node (hubert) where it resolves the external ip (from an
>> external dns) instead
>> of the internal ip (from the hosts file). thus one direction fails
>> (no external connection
>> at the moment for all but the master) and the other causes a lockup
>>
>> I hope that the explenation is not too convoluted
>> _______________________________________________
>> users mailing list
>> users_at_[hidden]
>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>
>