Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: Re: [OMPI users] Information about multi-path on IB-based systems
From: Jeff Squyres (jsquyres_at_[hidden])
Date: 2008-01-28 14:58:06

On Jan 23, 2008, at 3:12 PM, David Gunter wrote:

> Do I need to do anything special to enable multi-path routing on
> InfiniBand networks? For example, are there command-line arguments to
> mpiexec or the like?

It depends on what you mean by multi-path routing. For each MPI peer
pair (e.g., procs A and B), do you want Open MPI to use multiple
routes for MPI traffic over the IB fabric?

If you set the LMC in your fabric to be greater than 0, your SM should
issue multiple LIDs for each active port, and create routes
accordingly. I don't know what OpenSM does, but the Cisco SM will
naturally try to make the routes be across different links. Open MPI
(>=v1.2.x) should automatically detect that your LMC is >0 and setup
as many connections between a pair of MPI peers as the LMC allows and
then round robin packets across all of them.

Did that make sense?

Jeff Squyres
Cisco Systems