Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: Re: [OMPI users] one more finding in openmpi-1.7.5a1
From: Ralph Castain (rhc_at_[hidden])
Date: 2014-02-14 05:14:59


Thanks - hit me in the middle of the night over here that we had missed some options, but nice to find you had also seen it. Slightly modified patch will be applied and brought over to 1.7.5

On Feb 13, 2014, at 10:16 PM, tmishima_at_[hidden] wrote:

>
>
>
> Please try attached patch - from r30723.
>
> (See attached file: patch.rmaps_base_frame.from_r30723)
>
> Tetsuya Mishima
>
>> Thanks for prompt help.
>> Could you please resent the patch as attachment which can be applied with
> "patch" command, my mail client messes long lines.
>>
>>
>> On Fri, Feb 14, 2014 at 7:40 AM, <tmishima_at_[hidden]>wrote:
>>
>>
>> Thanks. I'm not familiar with mindist mapper. But obviously
>> checking for ORTE_MAPPING_BYDIST is missing. In addition,
>> ORTE_MAPPING_PPR is missing again by my mistake.
>>
>> Please try this patch.
>>
>> if OPAL_HAVE_HWLOC
>> } else if (ORTE_MAPPING_BYCORE == ORTE_GET_MAPPING_POLICY
>> (mapping)) {
>> ORTE_SET_RANKING_POLICY(tmp, ORTE_RANK_BY_CORE);
>> } else if (ORTE_MAPPING_BYL1CACHE == ORTE_GET_MAPPING_POLICY
>> (mapping)) {
>> ORTE_SET_RANKING_POLICY(tmp, ORTE_RANK_BY_L1CACHE);
>> } else if (ORTE_MAPPING_BYL2CACHE == ORTE_GET_MAPPING_POLICY
>> (mapping)) {
>> ORTE_SET_RANKING_POLICY(tmp, ORTE_RANK_BY_L2CACHE);
>> } else if (ORTE_MAPPING_BYL3CACHE == ORTE_GET_MAPPING_POLICY
>> (mapping)) {
>> ORTE_SET_RANKING_POLICY(tmp, ORTE_RANK_BY_L3CACHE);
>> } else if (ORTE_MAPPING_BYSOCKET == ORTE_GET_MAPPING_POLICY
>> (mapping)) {
>> ORTE_SET_RANKING_POLICY(tmp, ORTE_RANK_BY_SOCKET);
>> } else if (ORTE_MAPPING_BYNUMA == ORTE_GET_MAPPING_POLICY
>> (mapping)) {
>> ORTE_SET_RANKING_POLICY(tmp, ORTE_RANK_BY_NUMA);
>> } else if (ORTE_MAPPING_BYBOARD == ORTE_GET_MAPPING_POLICY
>> (mapping)) {
>> ORTE_SET_RANKING_POLICY(tmp, ORTE_RANK_BY_BOARD);
>> } else if (ORTE_MAPPING_BYHWTHREAD == ORTE_GET_MAPPING_POLICY
>> (mapping)) {
>> ORTE_SET_RANKING_POLICY(tmp, ORTE_RANK_BY_HWTHREAD);
>> } else if (ORTE_MAPPING_PPR == ORTE_GET_MAPPING_POLICY
>> (mapping)) {
>> ORTE_SET_RANKING_POLICY(tmp, ORTE_RANK_BY_SLOT);
>> } else if (ORTE_MAPPING_BYDIST == ORTE_GET_MAPPING_POLICY
>> (mapping)) {
>> ORTE_SET_RANKING_POLICY(tmp, ORTE_RANK_BY_SLOT);
>> #endif
>>
>> Tetsuya Mishima
>>
>>> Hi,
>>> after this patch we get this in jenkins:
>>>
>>> 07:03:15 [vegas12.mtr.labs.mlnx:01646] [[26922,0],0] ORTE_ERROR_LOG:
> Not
>> implemented in file rmaps_mindist_module.c at line 39107:03:15
>> [vegas12.mtr.labs.mlnx:01646] [[26922,0],0] ORTE_ERROR_LOG: Not
>>> implemented in file base/rmaps_base_map_job.c at line 285
>>>
>>>
>>>
>>>
>>>
>>> On Fri, Feb 14, 2014 at 6:35 AM, <tmishima_at_[hidden]>wrote:
>>>
>>>
>>> Sorry, one more shot - byslot was dropped!
>>>
>>> if (NULL == spec) {
>>> /* check for map-by object directives - we set the
>>> * ranking to match if one was given
>>> */
>>> if (ORTE_MAPPING_GIVEN & ORTE_GET_MAPPING_DIRECTIVE(mapping)) {
>>> if (ORTE_MAPPING_BYSLOT == ORTE_GET_MAPPING_POLICY
> (mapping))
>> {
>>> ORTE_SET_RANKING_POLICY(tmp, ORTE_RANK_BY_SLOT);
>>> } else if (ORTE_MAPPING_BYNODE == ORTE_GET_MAPPING_POLICY
>>> (mapping)) {
>>> ORTE_SET_RANKING_POLICY(tmp, ORTE_RANK_BY_NODE);
>>> #if OPAL_HAVE_HWLOC
>>> } else if (ORTE_MAPPING_BYCORE == ORTE_GET_MAPPING_POLICY
>>> (mapping)) {
>>> ORTE_SET_RANKING_POLICY(tmp, ORTE_RANK_BY_CORE);
>>> } else if (ORTE_MAPPING_BYL1CACHE ==
> ORTE_GET_MAPPING_POLICY
>>> (mapping)) {
>>> ORTE_SET_RANKING_POLICY(tmp, ORTE_RANK_BY_L1CACHE);
>>> } else if (ORTE_MAPPING_BYL2CACHE ==
> ORTE_GET_MAPPING_POLICY
>>> (mapping)) {
>>> ORTE_SET_RANKING_POLICY(tmp, ORTE_RANK_BY_L2CACHE);
>>> } else if (ORTE_MAPPING_BYL3CACHE ==
> ORTE_GET_MAPPING_POLICY
>>> (mapping)) {
>>> ORTE_SET_RANKING_POLICY(tmp, ORTE_RANK_BY_L3CACHE);
>>> } else if (ORTE_MAPPING_BYSOCKET == ORTE_GET_MAPPING_POLICY
>>> (mapping)) {
>>> ORTE_SET_RANKING_POLICY(tmp, ORTE_RANK_BY_SOCKET);
>>> } else if (ORTE_MAPPING_BYNUMA == ORTE_GET_MAPPING_POLICY
>>> (mapping)) {
>>> ORTE_SET_RANKING_POLICY(tmp, ORTE_RANK_BY_NUMA);
>>> } else if (ORTE_MAPPING_BYBOARD == ORTE_GET_MAPPING_POLICY
>>> (mapping)) {
>>> ORTE_SET_RANKING_POLICY(tmp, ORTE_RANK_BY_BOARD);
>>> } else if (ORTE_MAPPING_BYHWTHREAD ==
> ORTE_GET_MAPPING_POLICY
>>> (mapping)) {
>>> ORTE_SET_RANKING_POLICY(tmp, ORTE_RANK_BY_HWTHREAD);
>>> #endif
>>>
>>> Tetusya Mishima
>>>
>>>> I've found it. Please add 2 lines(770, 771) in rmaps_base_frame.c:
>>>>
>>>> 747 if (NULL == spec) {
>>>> 748 /* check for map-by object directives - we set the
>>>> 749 * ranking to match if one was given
>>>> 750 */
>>>> 751 if (ORTE_MAPPING_GIVEN & ORTE_GET_MAPPING_DIRECTIVE
>>>> (mapping)) {
>>>> 752 if (ORTE_MAPPING_BYCORE == ORTE_GET_MAPPING_POLICY
>>>> (mapping)) {
>>>> 753 ORTE_SET_RANKING_POLICY(tmp, ORTE_RANK_BY_CORE);
>>>> 754 } else if (ORTE_MAPPING_BYNODE ==
>>>> ORTE_GET_MAPPING_POLICY(mapping)) {
>>>> 755 ORTE_SET_RANKING_POLICY(tmp, ORTE_RANK_BY_NODE);
>>>> 756 } else if (ORTE_MAPPING_BYL1CACHE ==
>>>> ORTE_GET_MAPPING_POLICY(mapping)) {
>>>> 757 ORTE_SET_RANKING_POLICY(tmp,
>> ORTE_RANK_BY_L1CACHE);
>>>> 758 } else if (ORTE_MAPPING_BYL2CACHE ==
>>>> ORTE_GET_MAPPING_POLICY(mapping)) {
>>>> 759 ORTE_SET_RANKING_POLICY(tmp,
>> ORTE_RANK_BY_L2CACHE);
>>>> 760 } else if (ORTE_MAPPING_BYL3CACHE ==
>>>> ORTE_GET_MAPPING_POLICY(mapping)) {
>>>> 761 ORTE_SET_RANKING_POLICY(tmp,
>> ORTE_RANK_BY_L3CACHE);
>>>> 762 } else if (ORTE_MAPPING_BYSOCKET ==
>>>> ORTE_GET_MAPPING_POLICY(mapping)) {
>>>> 763 ORTE_SET_RANKING_POLICY(tmp,
> ORTE_RANK_BY_SOCKET);
>>>> 764 } else if (ORTE_MAPPING_BYNUMA ==
>>>> ORTE_GET_MAPPING_POLICY(mapping)) {
>>>> 765 ORTE_SET_RANKING_POLICY(tmp, ORTE_RANK_BY_NUMA);
>>>> 766 } else if (ORTE_MAPPING_BYBOARD ==
>>>> ORTE_GET_MAPPING_POLICY(mapping)) {
>>>> 767 ORTE_SET_RANKING_POLICY(tmp,
> ORTE_RANK_BY_BOARD);
>>>> 768 } else if (ORTE_MAPPING_BYHWTHREAD ==
>>>> ORTE_GET_MAPPING_POLICY(mapping)) {
>>>> 769 ORTE_SET_RANKING_POLICY(tmp,
>>>> ORTE_RANK_BY_HWTHREAD);
>>>> 770 } else if (ORTE_MAPPING_PPR ==
> ORTE_GET_MAPPING_POLICY
>>>> (mapping)) {
>>>> 771 ORTE_SET_RANKING_POLICY(tmp, ORTE_RANK_BY_SLOT);
>>>> 772 }
>>>>
>>>> Tetsuya Mishima
>>>>
>>>>> You are welcome, Ralph.
>>>>>
>>>>> But, after fixing it, I'm facing another problem whin I use ppr
>> option:
>>>>> [mishima_at_manage openmpi-1.7.4]$ mpirun -np 2 -map-by ppr:1:socket
>>>> -bind-to
>>>>> socket -report-bindings ~/mis/openmpi/demos/m
>>>>> yprog
>>>>> [manage.cluster:28057] [[25570,0],0] ORTE_ERROR_LOG: Not
> implemented
>> in
>>>>> file rmaps_ppr.c at line 389
>>>>> [manage.cluster:28057] [[25570,0],0] ORTE_ERROR_LOG: Not
> implemented
>> in
>>>>> file base/rmaps_base_map_job.c at line 285
>>>>>
>>>>> I confirmed it worked when it reverted back.
>>>>> I'm a little bit confused. Could you take a look?
>>>>>
>>>>> Tetsuya Mishima
>>>>>
>>>>>> Thanks - these used to be bitmaps, but changed when we started
>>> getting
>>>> so
>>>>> many options. Sadly, they are very rarely used, so bugs like this
> can
>>> go
>>>>> unnoticed for long times. Appreciate you taking such
>>>>>> a close look at them.
>>>>>>
>>>>>> Ralph
>>>>>>
>>>>>> On Feb 13, 2014, at 4:55 PM, tmishima_at_[hidden] wrote:
>>>>>>
>>>>>>>
>>>>>>> Hi Ralph,
>>>>>>>
>>>>>>> I would report one more finding in openmpi-1.7.5a1.
>>>>>>>
>>>>>>> Because ORTE_MAPPING_BY...s are not a bit field expression,
>>>>>>> at orte_rmaps_base_set_ranking_policy in rmaps_base_frame.c
>>>>>>> you should not use "&" to compare them:
>>>>>>>
>>>>>>> 747 if (NULL == spec) {
>>>>>>> 748 /* check for map-by object directives - we set
> the
>>>>>>> 749 * ranking to match if one was given
>>>>>>> 750 */
>>>>>>> 751 if (ORTE_MAPPING_GIVEN &
>> ORTE_GET_MAPPING_DIRECTIVE
>>>>>>> (mapping)) {
>>>>>>> 752 if (ORTE_MAPPING_BYCORE ==
>>> ORTE_GET_MAPPING_POLICY
>>>>>>> (mapping)) {
>>>>>>> 753 ORTE_SET_RANKING_POLICY(tmp,
>>>> ORTE_RANK_BY_CORE);
>>>>>>> 754 } else if (ORTE_MAPPING_BYNODE ==
>>>>>>> ORTE_GET_MAPPING_POLICY(mapping)) {
>>>>>>> 755 ORTE_SET_RANKING_POLICY(tmp,
>>>> ORTE_RANK_BY_NODE);
>>>>>>> 756 } else if (ORTE_MAPPING_BYL1CACHE ==
>>>>>>> ORTE_GET_MAPPING_POLICY(mapping)) {
>>>>>>> 757 ORTE_SET_RANKING_POLICY(tmp,
>>>>> ORTE_RANK_BY_L1CACHE);
>>>>>>> 758 } else if (ORTE_MAPPING_BYL2CACHE ==
>>>>>>> ORTE_GET_MAPPING_POLICY(mapping)) {
>>>>>>> 759 ORTE_SET_RANKING_POLICY(tmp,
>>>>> ORTE_RANK_BY_L2CACHE);
>>>>>>> 760 } else if (ORTE_MAPPING_BYL3CACHE ==
>>>>>>> ORTE_GET_MAPPING_POLICY(mapping)) {
>>>>>>> 761 ORTE_SET_RANKING_POLICY(tmp,
>>>>> ORTE_RANK_BY_L3CACHE);
>>>>>>> 762 } else if (ORTE_MAPPING_BYSOCKET ==
>>>>>>> ORTE_GET_MAPPING_POLICY(mapping)) {
>>>>>>> 763 ORTE_SET_RANKING_POLICY(tmp,
>>>>> ORTE_RANK_BY_SOCKET);
>>>>>>> 764 } else if (ORTE_MAPPING_BYNUMA ==
>>>>>>> ORTE_GET_MAPPING_POLICY(mapping)) {
>>>>>>> 765 ORTE_SET_RANKING_POLICY(tmp,
>>>> ORTE_RANK_BY_NUMA);
>>>>>>> 766 } else if (ORTE_MAPPING_BYBOARD ==
>>>>>>> ORTE_GET_MAPPING_POLICY(mapping)) {
>>>>>>> 767 ORTE_SET_RANKING_POLICY(tmp,
>>>>> ORTE_RANK_BY_BOARD);
>>>>>>> 768 } else if (ORTE_MAPPING_BYHWTHREAD ==
>>>>>>> ORTE_GET_MAPPING_POLICY(mapping)) {
>>>>>>> 769 ORTE_SET_RANKING_POLICY(tmp,
>>>>>>> ORTE_RANK_BY_HWTHREAD);
>>>>>>> 770 }
>>>>>>>
>>>>>>> Regards,
>>>>>>> Tetsuya Mishima
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> users mailing list
>>>>>>> users_at_[hidden]
>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>>>
>>>>>> _______________________________________________
>>>>>> users mailing list
>>>>>> users_at_[hidden]
>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>>
>>>>> _______________________________________________
>>>>> users mailing list
>>>>> users_at_[hidden]
>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>
>>>> _______________________________________________
>>>> users mailing list
>>>> users_at_[hidden]
>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>
>>> _______________________________________________
>>> users mailing list
>>> users_at_[hidden]
>>>
>>
> http://www.open-mpi.org/mailman/listinfo.cgi/users_______________________________________________
>
>>
>>> users mailing list
>>> users_at_[hidden]http://www.open-mpi.org/mailman/listinfo.cgi/users
>>
>> _______________________________________________
>> users mailing list
>> users_at_[hidden]
>>
> http://www.open-mpi.org/mailman/listinfo.cgi/users_______________________________________________
>
>> users mailing list
>> users_at_[hidden]http://www.open-mpi.org/mailman/listinfo.cgi/users
> <patch.rmaps_base_frame.from_r30723>_______________________________________________
> users mailing list
> users_at_[hidden]
> http://www.open-mpi.org/mailman/listinfo.cgi/users