Open MPI logo

Hardware Locality Users' Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Hardware Locality Users mailing list

Subject: Re: [hwloc-users] Finding closest host bridge
From: Brice Goglin (Brice.Goglin_at_[hidden])
Date: 2014-01-28 17:29:43

The bridge cannot be "not connected to anything". All objects have a
parent (and are a child of that parent) except the very-top root object.
Theoretically, the bridge could be connected anywhere. In practice it's
connected to a NUMA node, a root object, or (rarely) a group of numa nodes.

The problem with your code is that it cannot work on machines where some
cores have no PCI bus nearby (and this happens for real). Ideally, you
would walk up until you find a bridge, but sometimes you will never find
any of them. For instance, the closest bridge could be attached to
another NUMA node close to yours on a large NUMA machine.

I assume you're not looking for a bridge in the end, but rather for PCI
devices such as GPUs or network interface? I would suggest reverting the
logic: Traverse the list of PCI devices (or bridges if you really care
about bridges), check if they are what you want, and check if your core
is near them. The locality of the PCI device (or bridge, or os device)
can be retrieved with hwloc_get_nonio_ancestor_obj() which returns the
object where the corresponding bus/bridge is attached.

Does that help?


Le 28/01/2014 23:12, Rolf vandeVaart a écrit :
> I cam up with this snippet of code so I could find the PCI devices closest to where a process was running. This works fine on a NUMA node. However, on a non-NUMA node, it appears that the HWLOC bridge is not connected to anything. Therefore, I believe I just need to do something like hwloc_get_obj_by_type(topology, HWLOC_OBJ_BRIDGE, 0) when running on a non-NUMA node. Does that seem correct? Should I special case where I do not find a HWLOC_OBJ_NODE object?
> retval = hwloc_topology_init(&topology);
> retval = hwloc_topology_set_flags(topology, flags);
> retval = hwloc_topology_load(topology);
> newset = hwloc_bitmap_alloc();
> retval = hwloc_get_last_cpu_location(topology, newset, 0);
> /* Get the object that contains the cpuset */
> node = hwloc_get_first_largest_obj_inside_cpuset(topology, newset);
> /* Climb up from that object until we find HWLOC_OBJ_NODE */
> while (node->type != HWLOC_OBJ_NODE) {
> node = node->parent;
> }
> /* Now look for the HWLOC_OBJ_BRIDGE. All PCI busses hanging off the
> * node will have one of these */
> bridge = hwloc_get_next_child(topology, node, NULL);
> while (bridge != NULL && bridge->type != HWLOC_OBJ_BRIDGE) {
> bridge = hwloc_get_next_child(topology, node, bridge);
> }
> -----------------------------------------------------------------------------------
> This email message is for the sole use of the intended recipient(s) and may contain
> confidential information. Any unauthorized review, use, disclosure or distribution
> is prohibited. If you are not the intended recipient, please contact the sender by
> reply email and destroy all copies of the original message.
> -----------------------------------------------------------------------------------
> _______________________________________________
> hwloc-users mailing list
> hwloc-users_at_[hidden]