Open MPI logo

Hardware Locality Development Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Hardware Locality Development mailing list

Subject: Re: [hwloc-devel] #23: network topology support and v1.0semanticfixes
From: Jeff Squyres (jsquyres_at_[hidden])
Date: 2010-01-11 14:21:53

On Jan 11, 2010, at 11:12 AM, Samuel Thibault wrote:

> > Are you thinking of adding bandwidth attributes?
> When the information is available through some way (be it by hand), yes.
> > Or are you thinking of adding weighting between objects in the hierarchy? Or ...?
> I'd tend to think that this would be an upper layer that applications
> would compute themselves, according to their needs.

FWIW, some networks provide this kind of information through an API, such as verbs. But it's not a straight "bandwidth" information; verbs provides lanes and widths. The point: different networks may report network-specific information differently.

More below.

> > More generally -- some objects can be bound to, some cannot.
> >
> > How does this kind of thing extend to, say, co-processors (such as accelerators, FPGAs, GPGPUs, etc.)?
> I once thought about adding gpuset, maybe, but since the kinds of
> objects will probably vary a lot, maybe it's better to not try to be
> smart and let layers above handle it, only possibly provide for each
> kind the OS number (e.g. CUDA device number).

Maybe it would be better to add a (void*) on to the object to allow arbitrary 3rd parties to cache information off any given hwloc object. This would allow the upper-layer to assign specific context (such as additional device information) to any given hwloc object that could even persist across tree copies, etc.

> > > - Helpers that take cpuset parameters of course don't make sense any more
> > > when applied to networked topologies. But it probably doesn't make
> > > sense for the caller to call them in the first place, and the caller
> > > knows it since it's the caller that has first called the combining
> > > operation or loaded an XML file resulting from it.
> >
> > Agreed. Perhaps we should have a general query function that can return whether a given object can be bound to or not (e.g., for generic tree-traversal kinds of functionality)...?
> Well, to be "bound to" here would mean for a thread, i.e. to be bound
> to a CPU set. So testing for cpuset != NULL should be enough.

Ok, fair enough. We should document that assumption somewhere.

> As said
> above, I'm not sure we necessarily want to extend that notion since
> the various kinds of "binding something to" becomes numerous.

Yep, agree.

> > > The plan I see is that for 1.0 we only check that catenating .XML files
> > > by hand to build misc levels representing network layers does indeed
> > > work, which should mean that actual combining functions etc. should be
> > > possible to implement later.
> >
> > FWIW, I'd prefer to see the combining/etc. functions ASAP -- we could definitely use such things in Open MPI...
> Mmm, doing it in the API is not so trivial, for 1.0 I'd rather just
> provide a small xml combiner (just a matter of tail/head/cat), which may
> be enough for a start. But if you need it at the API level, we can add
> it to the 1.0 milestone.

K. "He who implements, wins". So perhaps I can volunteer to implement this stuff. :-)

I'm curious -- what's the definition of cat'ing 2 XML files together? Does the 2nd become a subtree of the first?

Jeff Squyres