Brock Palen wrote:
> has anyone done work with hwloc on scalemp systems? They provide
> their own tool numabind, but we are looking for a more generic
> solution to process placement and control that works well inside our
> MPI library (openMPI in most cases).
> Any input on this would be great!
>From what I remember, ScaleMP uses an hypervisor on each node that
virtually merges all of them into a fake big shared-memory machine. Then
a vanilla Linux kernel runs on top of it. So hwloc should just see
regular cores and NUMA node information, assuming the virtual "merged"
hardware reports all necessary information to the OS.
There's a bit of ScaleMP code in the Linux kernel, but it does pretty
much nothing, it does not seem to add anything to /proc or /sys for
instance. So I am not sure hwloc could get some specialized knowledge of
ScaleMP machines. Maybe their custom numabind tool knows that ScaleMP
machines only works on machines with some well-defined
types/counts/numbering of processors and NUMA nodes, and thus uses this
information to group sockets/NUMA-nodes depending on their physical