Open MPI logo

Hardware Locality Development Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Hardware Locality Development mailing list

Subject: Re: [hwloc-devel] Support for new architecture
From: Brice Goglin (Brice.Goglin_at_[hidden])
Date: 2013-11-09 03:38:10


Hello Ralph,

Is there something below that is the main reason of this mail? I don't
anything obviously incompatible with hwloc here. This all seems mostly
hierarchical to me, nothing harder than a NUMA machine with a strange
NUMA interconnect.

A couple notes:
* NIC don't require a PCI bus in hwloc. We can attach OS devices
everywhere. And we can add other types of buses if needed (the bridge
type is already configurable).
* Your hierarchy of objects may have to use custom types or generic Groups
* if there are independent lightweight kernels on small parts of the
machine, we'll have to be careful assuming things like binding don't
work across different subparts
* the same controller in "common support infrastructure" in host or
boards could be individual NICs, one for each subcomponent?
* if there are objects within a single parent that have different
distances to that common infrastructure, there's still a way to do it
(put the NIC under a group of nearby sockets, and add a distance matrix
explaining all this).

If that doesn't work, we'll do a hwloc v2.0 with very intrusive changes,
we'll see :)

Brice

Le 08/11/2013 17:42, Ralph Castain a écrit :
> Hi folks
>
> We are seeing a new architecture appearing in the very near future, and I'm not sure how hwloc will handle it. Consider the following case:
>
> * I have a rack that contains multiple "hosts"
>
> * each host consists of a box/shelf with common support infrastructure in it - it has some kind of controller in it, and might have some networking support, maybe a pool of memory that can be allocated across the occupants.
>
> * in the host, I have one or more "boards". Each board again has a controller in it with some common infrastructure to support its local sockets - might include some networking that would look like NICs (though not necessarily on a PCIe interface), a board-level memory pool, etc.
>
> * each socket contains one or more die. Each die runs its own instance of an OS - probably a lightweight kernel - that can vary between dies (e.g., might have a tweaked configuration), and has its own associated memory that will physically reside outside the socket. You can think of each die as constituting a "shared memory locus" - i.e., processes running on that die can share memory between them as it would sit under the same OS instance.
>
> * each die has some number of cores/hwthreads/caches etc.
>
> Note that the sockets are not sitting in some PCIe bus - they appear to be directly connected to the overall network just like a "node" would appear today. However, there is a definite need for higher layers (RMs and MPIs) to understand this overall hierarchy and the "distances" between the individual elements.
>
> Any thoughts on how we can support this?
> Ralph
>
> _______________________________________________
> hwloc-devel mailing list
> hwloc-devel_at_[hidden]
> http://www.open-mpi.org/mailman/listinfo.cgi/hwloc-devel