Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: Re: [OMPI users] Heterogeneous OpenFabrics hardware
From: Peter Kjellstrom (cap_at_[hidden])
Date: 2009-01-27 06:08:32


On Monday 26 January 2009, Jeff Squyres wrote:
> The Interop Working Group (IWG) of the OpenFabrics Alliance asked me
> to bring a question to the Open MPI user and developer communities: is
> anyone interested in having a single MPI job span HCAs or RNICs from
> multiple vendors? (pardon the cross-posting, but I did want to ask
> each group separately -- because the answers may be different)
>
> The interop testing lab at the University of New Hampshire
> (http://www.iol.unh.edu/services/testing/ofa/ ) discovered that most (all?)
> MPI implementations fail when having a single MPI job span HCAs from
> multiple vendors and/or span RNICs from multiple vendors. I don't remember
> the exact details (and they may not be public, anyway), but I'm pretty sure
> that OMPI failed when used with QLogic and Mellanox HCAs in a single MPI
> job. This is fairly unsurprising, given how we tune Open MPI's use of
> OpenFabrics-capable hardware based on our .ini file.
>
> So my question is: does anyone want/need to support jobs that span
> HCAs from multiple vendors and/or RNICs from multiple vendors?

For these three cases:

1) Different vedor id but same OFED driver and basic chip
2) Same chip vendor, different OFED driver (mthca vs mlx4)
3) Any OFED supported IB HCA

IMHO:

Number one should just work. We may at times have some nodes with HCAs that
have been flashed with non-standard/non-vendor firmware.

Number two is something I would kind of expect to work. A possible situation
where I'd need it is if I temporarily use an older HCA (mthca) to get a node
going on a cluster with ConnectX (mlx4). Another case could be a cluster with
two partitions with different HCAs.

Number three would be nice to have. I think many users would assume it to
work. Why not? They have symmetric software, all nodes run OFED, all have
working IB... It would have worked if their nodes had had different kinds of
ethernet NICS...

/Peter