Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: Re: [OMPI users] multi-rail failover with IB
From: Jeff Squyres (jsquyres_at_[hidden])
Date: 2008-04-02 15:36:50

On Apr 2, 2008, at 1:13 AM, Robin Humble wrote:

> from reading the FAQ and this list it seems OpenMPI can use multiple
> InfiniBand rails by round-robining across the ports out of each node
> (as
> long as they're configured to be on separate subnets (I think)).

They can handle it if they're on the same subnet, too. The FAQ is
trying to say that if you *do* have multiple different air-gapped
subnets, they must have different subnet ID's so that OMPI can compute
reachability correctly.

> can OpenMPI also deal with one of the subnets failing?
> ie. will OpenMPI automatically fall back to using the last remaining
> working IB port out of a node, or even fallback to GigE if all the IB
> fails?

Not in the 1.2 series.

The 1.3 series *may* include "APM" support (automatic path migration
-- a feature in IB). It looks positive that that'll make the 1.3 cut,
but I don't have definite information yet.

> the reason I ask is that we're worried about switches failing in the
> IB
> network and whether OpenMPI can solve all our problems for us if we
> configure up 2 or more independent IB networks out of each node.

Not for the 1.2 series, sorry. If APM makes it into the v1.3 series,
then yes, that'll work.

> possibly this sort of failover in the MPI isn't needed with ConnectX
> as
> long as it's adaptive routing works as advertised? If so then I guess
> it's not that important, and I wouldn't want to make you guys do a lot
> of unecessary work :-)
> the FAQ entry here:
> says
> - Data Reliability and network fault tolerance. Similar to those
> implemented in LA-MPI
> but I don't actually know what LA-MPI implemented in this area, so
> that
> doesn't really help me.

That stuff was *started* in OMPI, but demand for it fell off, so no
one really completed it.

Jeff Squyres
Cisco Systems