On Jan 3, 2008, at 9:03 AM, Gleb Natapov wrote:
> In Paris we've talked about putting HCA discovery and
> initialization code
> outside of openib BTL so other components that want to use IB will
> be able
> to share common code, data and registration cache. Other components
> I am
> thinking about are ofud and multicast collectives. I started to look
> at
> this and I have a couple of problems with this approach. Currently
> openib
> BTL has if_include/if_exclude parameters to control which HCAs
> should be
> used. Should we make those parameters global and initialize only HCAs
> that are not exulted by those filters, or should we initialize all
> HCAs
> and each component will have its own include/exclude filters?
Good question. I think the optimal solution would be to have one set
of globals (common_of_if_include or somesuch?) with optional per-
component overrides. E.g., tell all of OMPI to if_include mthca0, but
then tell just the multicast collectives to if_include ipath1 (for
whatever reason). This would allow fine-grained selection of which
communication types use which devices.
To minimize the repetition of code, this could be effected by having a
function in the common/of area that does all the work for the include/
exclude behavior. You can simply call it with any of the MCA param
values, such as: common_of_if_in/exclude, btl_openib_if_in/exclude,
coll_of_if_in/exclude, ... and it can return a list of ports to use.
> Another
> problem is how multicast collective knows that all processes in a
> communicator are reachable via the same network, do we have a
> mechanism
> in ompi to check this?
Good question.
Perhaps the common_of stuff could hang some data off the ompi_proc_t
that can be read by any of-like component (btl openib, coll of
multicast, etc.)...? This could contain a subnet ID, or perhaps a
reachable flag, or somesuch.
--
Jeff Squyres
Cisco Systems
|
![[Flag]](../../../../../community/mirrors/flag-us.gif){kind=small}