Open MPI logo

Open MPI Development Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Development mailing list

Subject: Re: [OMPI devel] two questions about 1.7.1
From: Ralph Castain (rhc_at_[hidden])
Date: 2013-11-22 13:41:38


I'm afraid Jeff is on vacation until Dec 2nd, Paul, so response will be delayed.

On Nov 22, 2013, at 10:19 AM, Paul Kapinos <kapinos_at_[hidden]> wrote:

> Hi Jeff,
>
> On 06/19/13 15:26, Jeff Squyres (jsquyres) wrote:
> ...
>>> II.
>>> In the 1.7.x series, the 'carto' framework has been deleted:
>>> http://www.open-mpi.org/community/lists/announce/2013/04/0053.php
>>>> - Removed maffinity, paffinity, and carto frameworks (and associated
>>>> MCA params).
>>>
>>> Is there some replacement for this? Or, would Open MPI detect the NUMA structure of nodes automatically?
>>
>> Yes. OMPI uses hwloc internally now to figure this stuff out.
>>
>>> Background: Currently we're using the 'carto' framework on our kinda special 'Bull BCS' nodes. Each such node consist of 4 boards with own IB card but build a shared memory system. Clearly, communicating should go over the nearest IB interface - for this we use 'carto' now.
>>
>> It should do this automatically in the 1.7 series.
>>
>> Hmm; I see there isn't any verbose output about which devices it picks, though. :-( Try this patch, and run with --mca btl_base_verbose 100 and see if you see appropriate devices being mapped to appropriate processes:
>>
>> Index: mca/btl/openib/btl_openib_component.c
>> ===================================================================
>> --- mca/btl/openib/btl_openib_component.c (revision 28652)
>> +++ mca/btl/openib/btl_openib_component.c (working copy)
>> @@ -2712,6 +2712,8 @@
>> mca_btl_openib_component.ib_num_btls <
>> mca_btl_openib_component.ib_max_btls); i++) {
>> if (distance != dev_sorted[i].distance) {
>> + BTL_VERBOSE(("openib: skipping device %s; it's too far away",
>> + ibv_get_device_name(dev_sorted[i].ib_dev)));
>> break;
>> }
>
> Well, I've tried this path on actual 1.7.3 (where the code is moved some 12 lines - beginning with 2700).
> !! - no output "skipping device"! Also when starting main processes and -bind-to-socket used. What I see is
> >[cluster.rz.RWTH-Aachen.DE:43670] btl:usnic: found: device mlx4_1, port 1
> >[cluster.rz.RWTH-Aachen.DE:43670] btl:usnic: this is not a usnic-capable device
> >[cluster.rz.RWTH-Aachen.DE:43670] btl:usnic: found: device mlx4_0, port 1
> >[cluster.rz.RWTH-Aachen.DE:43670] btl:usnic: this is not a usnic-capable device
> .. one message block per process. Is seems that processes see both IB cards in the special nodes(*) but none were disabled, or at least the verbosity path did not worked.
>
> Well, is there any progress on this frontline? Or, can I activate more verbosity / what did I do wrong with the path? (see attached file)
>
> Best!
> Paul Kapinos
>
>
> *) the nodes used for testing are also Bull BCS nodes but vonsisting of just two boards instead of 4
> --
> Dipl.-Inform. Paul Kapinos - High Performance Computing,
> RWTH Aachen University, Center for Computing and Communication
> Seffenter Weg 23, D 52074 Aachen (Germany)
> Tel: +49 241/80-24915
> <btl_openib_component.c>_______________________________________________
> devel mailing list
> devel_at_[hidden]
> http://www.open-mpi.org/mailman/listinfo.cgi/devel