Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: Re: [OMPI users] Hanging vs Stopping behaviour in communication failures
From: Jeff Squyres (jsquyres_at_[hidden])
Date: 2009-12-14 15:29:46


On Dec 9, 2009, at 3:47 AM, Constantinos Makassikis wrote:

> sometimes when running Open MPI jobs, the application hangs. By looking the
> output I get the following error message:
>
> [ic17][[34562,1],74][../../../../../ompi/mca/btl/tcp/btl_tcp_frag.c:216:mca_btl_tcp_frag_recv
> ] mca_btl_tcp_frag_recv: readv failed: No route to host (113)
>
> I would expect Open MPI to eventually quit with an error at such situations.
> Is the observed behaviour (i.e.: hanging) the intended one ?

That does seem weird. I would think that we should abort rather than hang. But then again, the code is fairly hairy there -- there are many corner cases.

> If so, what would be the reason(s) behind choosing the hanging over the
> stopping ?

It *looks* like the code is supposed to retry the connection here, but perhaps something is not operating correctly (or perhaps it *is* trying to reconnect and the network is failing to reconnect for some reason...?).

How often does this happen? Is it in the middle of the application run, or at the very beginning? Do you have any other network issues where connections get dropped, etc.? Do you have any firewalls running on your cluster machines?

-- 
Jeff Squyres
jsquyres_at_[hidden]