Open MPI logo

Hardware Locality Users' Mailing List Archives

  |   Home   |   Support   |   FAQ   |  

This web mail archive is frozen.

This page is part of a frozen web archive of this mailing list.

You can still navigate around this archive, but know that no new mails have been added to it since July of 2016.

Click here to be taken to the new web archives of this list; it includes all the mails that are in this frozen archive plus all new mails that have been sent to the list since it was migrated to the new archives.

Subject: Re: [hwloc-users] Using distances
From: Jeffrey Squyres (jsquyres_at_[hidden])
Date: 2012-04-21 09:14:06


Got it. Thanks!

On Apr 21, 2012, at 8:05 AM, Brice Goglin wrote:

> On 21/04/2012 13:15, Jeffrey Squyres wrote:
>> On Apr 21, 2012, at 7:09 AM, Brice Goglin wrote:
>>
>>> I assume you have the entire distance (latency) matrix between all NUMA nodes as usually reported by the BIOS.
>>>
>>> const struct hwloc_distance_s *distances = hwloc_get_whole_distance_matrix_by_type(topology, HWLOC_OBJ_NODE);
>>> assert(distances);
>>> assert(distances->latency);
>> Is this stored on the topology object?
>
> No it's stored in the object that covers all objects connected by the distance matrix. So usually in the root object.
>
>> Hence, if this distance data is already covered by the XML export/import, then I should have this data.
>
> Yes it should be there.
>
>>> Now distances->latency[a+b*distances->nbobjs] contains the latency between NUMA nodes whose *logical* indexes are a and b (it may be asymmetrical).
>>>
>>> Now get the NUMA node object close to your PUs and the NUMA objects close to each OFED device, take their ->logical_index and you'll get the latencies.
>> Ah, ok. This is what I didn't understand from the docs -- is there no distance to actual PCI devices? I.e., distance is only measured between NUMA nodes?
>>
>> I ask because the functions allow measuring distance by depth and type -- are those effectively ignored, and really all you can check is the distance between NUMA nodes?
>
> You can have distance matrices between any object sets of any type/depth. Depends what the BIOS reports or what the user adds. The BIOS usually only reports NUMA node distances.
>
> We could extend them by saying that the distance between any child of NUMA node X and any child of NUMA node Y is equal to the distance between NUMA nodes X and Y, but we don't do that.
>
> One reason is that the current distance stuff lets the user add a distance matrix between NUMA nodes and another one between sockets, even if they are incompatible. When this happens, which one do you use to generate the distance between two cores?
>
> There are some tickets in trac they will help clarifying all this mess.
>
> Brice
>
>
> _______________________________________________
> hwloc-users mailing list
> hwloc-users_at_[hidden]
> http://www.open-mpi.org/mailman/listinfo.cgi/hwloc-users

-- 
Jeff Squyres
jsquyres_at_[hidden]
For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/