Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |  

This web mail archive is frozen.

This page is part of a frozen web archive of this mailing list.

You can still navigate around this archive, but know that no new mails have been added to it since July of 2016.

Click here to be taken to the new web archives of this list; it includes all the mails that are in this frozen archive plus all new mails that have been sent to the list since it was migrated to the new archives.

Subject: Re: [OMPI users] Hanging vs Stopping behaviour in communication failures
From: Jeff Squyres (jsquyres_at_[hidden])
Date: 2009-12-14 15:29:46


On Dec 9, 2009, at 3:47 AM, Constantinos Makassikis wrote:

> sometimes when running Open MPI jobs, the application hangs. By looking the
> output I get the following error message:
>
> [ic17][[34562,1],74][../../../../../ompi/mca/btl/tcp/btl_tcp_frag.c:216:mca_btl_tcp_frag_recv
> ] mca_btl_tcp_frag_recv: readv failed: No route to host (113)
>
> I would expect Open MPI to eventually quit with an error at such situations.
> Is the observed behaviour (i.e.: hanging) the intended one ?

That does seem weird. I would think that we should abort rather than hang. But then again, the code is fairly hairy there -- there are many corner cases.

> If so, what would be the reason(s) behind choosing the hanging over the
> stopping ?

It *looks* like the code is supposed to retry the connection here, but perhaps something is not operating correctly (or perhaps it *is* trying to reconnect and the network is failing to reconnect for some reason...?).

How often does this happen? Is it in the middle of the application run, or at the very beginning? Do you have any other network issues where connections get dropped, etc.? Do you have any firewalls running on your cluster machines?

-- 
Jeff Squyres
jsquyres_at_[hidden]