Hi Jeff,

Sorry for the delay, but my victim with 2 ib devices had been stolen ;-)

So, I ported the patch on the v1.5 branch and finally could test it.

Actually, there is no opal_hwloc_base_get_topology() in v1.5 so I had to set
the hwloc flags in ompi_mpi_init() and orte_odls_base_open() (i.e. the places
where opal_hwloc_topology is initialized).

With the new flag set, hwloc_get_nbobjs_by_type(opal_hwloc_topology, HWLOC_OBJ_CORE)
is now seeing the actual number of cores on the node (instead of 1 when our
cpuset is a singleton).

Since opal_paffinity_base_get_processor_info() calls module_get_processor_info()
(in hwloc/paffinity_hwloc_module.c), which in turn calls hwloc_get_nbobjs_by_type(),
we are now getting the right number of cores in get_ib_dev_distance().

So we are looping over the exact number of cores, looking for a potential binding.

So as a conclusion, there's no need for any other patch: the fix you committed
was the only one needed to fix the issue.

Could you please move it to v1.5 (do I need to fill a CMR)?

Thanks!

 
--
Nadia Derbey
 

devel-bounces@open-mpi.org wrote on 02/09/2012 06:00:48 PM:

> De : Jeff Squyres <jsquyres@cisco.com>
> A : Open MPI Developers <devel@open-mpi.org>
> Date : 02/09/2012 06:01 PM
> Objet : Re: [OMPI devel] btl/openib: get_ib_dev_distance doesn't see
> processes as bound if the job has been launched by srun
> Envoyé par : devel-bounces@open-mpi.org
>
> Nadia --
>
> I committed the fix in the trunk to use HWLOC_WHOLE_SYSTEM and IO_DEVICES.
>
> Do you want to revise your patch to use hwloc APIs with
> opal_hwloc_topology (instead of paffinity)?  We could use that as a
> basis for the other places you identified that are doing similar things.
>
>
> On Feb 9, 2012, at 8:34 AM, Ralph Castain wrote:
>
> > Ah, okay - in that case, having the I/O device attached to the
> "closest" object at each depth would be ideal from an OMPI perspective.
> >
> > On Feb 9, 2012, at 6:30 AM, Brice Goglin wrote:
> >
> >> The bios usually tells you which numa location is close to each
> host-to-pci bridge. So the answer is yes.
> >> Brice
> >>
> >>
> >> Ralph Castain <rhc@open-mpi.org> a écrit :
> >> I'm not sure I understand this comment. A PCI device is attached
> to the node, not to any specific location within the node, isn't it?
> Can you really say that a PCI device is "attached" to a specific
> NUMA location, for example?
> >>
> >>
> >> On Feb 9, 2012, at 6:15 AM, Jeff Squyres wrote:
> >>
> >>> That doesn't seem too attractive from an OMPI perspective,
> though.  We'd want to know where the PCI devices are actually rooted.
> >>
> >> _______________________________________________
> >> devel mailing list
> >> devel@open-mpi.org
> >> http://www.open-mpi.org/mailman/listinfo.cgi/devel
> >
> > _______________________________________________
> > devel mailing list
> > devel@open-mpi.org
> > http://www.open-mpi.org/mailman/listinfo.cgi/devel
>
>
> --
> Jeff Squyres
> jsquyres@cisco.com
> For corporate legal information go to:
> http://www.cisco.com/web/about/doing_business/legal/cri/
>
>
> _______________________________________________
> devel mailing list
> devel@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/devel