Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: Re: [OMPI users] Ensuring use of real cores
From: Jeff Squyres (jsquyres_at_[hidden])
Date: 2012-09-12 09:50:39


A little more on this (since affinity is one of my favorite topics of late :-) ). See my blog entries about what we just did in the 1.7 branch (and SVN trunk):

http://blogs.cisco.com/performance/taking-mpi-process-affinity-to-the-next-level/
http://blogs.cisco.com/performance/process-affinity-in-ompi-v1-7-part-1/
http://blogs.cisco.com/performance/process-affinity-in-ompi-v1-7-part-2/

As Ralph said, the v1.6 series will allow you to bind processes to the entire core (i.e., all hyperthreads in a core), or to an entire socket (i.e., all hyperthreads in a socket).

The v1.7 series will be quite a bit more flexible in its affinity options (note that the "Expert" mode described in my blog posting will be coming in v1.7.1 -- if you want to try that now, you'll need to use the SVN trunk).

For example:

mpirun --report-bindings --map-by core --bind-to hwthread ...

Should give you the pattern you want. Note that it looks like we have a bug in this pattern at the moment, however -- you'll need to use the SVN trunk and the "lama" mapper to get the patterns you want. The following example is running on a sandy bridge server with 2 sockets, each with 8 cores, each with 2 hyperthreads:

One hyperthread per core:

-----
% mpirun --mca rmaps lama -np 4 --host svbu-mpi058 --report-bindings --map-by core --bind-to hwthread uptime
[svbu-mpi058:23916] MCW rank 0 bound to socket 0[core 0[hwt 0]]: [B./../../../../../../..][../../../../../../../..]
[svbu-mpi058:23916] MCW rank 1 bound to socket 0[core 1[hwt 0]]: [../B./../../../../../..][../../../../../../../..]
[svbu-mpi058:23916] MCW rank 2 bound to socket 0[core 2[hwt 0]]: [../../B./../../../../..][../../../../../../../..]
[svbu-mpi058:23916] MCW rank 3 bound to socket 0[core 3[hwt 0]]: [../../../B./../../../..][../../../../../../../..]
 06:48:51 up 1 day, 12:08, 0 users, load average: 0.00, 0.01, 0.05
 06:48:51 up 1 day, 12:08, 0 users, load average: 0.00, 0.01, 0.05
 06:48:51 up 1 day, 12:08, 0 users, load average: 0.00, 0.01, 0.05
 06:48:51 up 1 day, 12:08, 0 users, load average: 0.00, 0.01, 0.05
-----

Both hyperthreads per core:

-----
% mpirun --mca rmaps lama -np 4 --host svbu-mpi058 --report-bindings --map-by core --bind-to core uptime
[svbu-mpi058:23951] MCW rank 0 bound to socket 0[core 0[hwt 0-1]]: [BB/../../../../../../..][../../../../../../../..]
[svbu-mpi058:23951] MCW rank 1 bound to socket 0[core 1[hwt 0-1]]: [../BB/../../../../../..][../../../../../../../..]
[svbu-mpi058:23951] MCW rank 2 bound to socket 0[core 2[hwt 0-1]]: [../../BB/../../../../..][../../../../../../../..]
[svbu-mpi058:23951] MCW rank 3 bound to socket 0[core 3[hwt 0-1]]: [../../../BB/../../../..][../../../../../../../..]
 06:48:57 up 1 day, 12:09, 0 users, load average: 0.00, 0.01, 0.05
 06:48:57 up 1 day, 12:09, 0 users, load average: 0.00, 0.01, 0.05
 06:48:57 up 1 day, 12:09, 0 users, load average: 0.00, 0.01, 0.05
 06:48:57 up 1 day, 12:09, 0 users, load average: 0.00, 0.01, 0.05
------

On Sep 12, 2012, at 8:10 AM, John R. Cary wrote:

> Thanks!
>
> John
>
> On 9/12/12 8:05 AM, Ralph Castain wrote:
>> On Sep 12, 2012, at 4:57 AM, "John R. Cary" <cary_at_[hidden]> wrote:
>>
>>> I do want in fact to bind first to one HT of each core
>>> before binding to two HTs of one core. So that will
>>> be possible in 1.7?
>> Yes - you can get a copy of the 1.7 nightly tarball and experiment with it in advance, if you like. You'll want
>>
>> mpirun --map-by core --bind-to hwthread ....
>>
>> Add --report-bindings to see what happens, but I believe that will do what you want. You'll map one process to each core, and bind it to only the first hwthread on that core.
>>
>> Let me know either way - if it doesn't, we have time to adjust it.
>>
>>> Thx....John
>>>
>>> On 9/11/12 11:19 PM, Ralph Castain wrote:
>>>> Not entirely sure I know what you mean. If you are talking about running without specifying binding, then it makes no difference - we'll run wherever the OS puts us, so you would need to tell the OS not to use the virtual cores (i.e., disable HT).
>>>>
>>>> If you are talking about binding, then pre-1.7 releases all bind to core at the lowest level. On a hyperthread-enabled machine, that binds you to both HT's of a core. Starting with the upcoming 1.7 release, you can bind to the separate HTs, but that doesn't sound like something you want to do.
>>>>
>>>> HTH
>>>> Ralph
>>>>
>>>>
>>>> On Sep 11, 2012, at 6:34 PM, John R. Cary <cary_at_[hidden]> wrote:
>>>>
>>>>> Our code gets little benefit from using virtual cores (hyperthreading),
>>>>> so when we run with mpiexec on an 8 real plus 8 virtual machine, we
>>>>> would like to be certain that it uses only the 8 real cores.
>>>>>
>>>>> Is there a way to do this with openmpi?
>>>>>
>>>>> Thx....John
>>>>> _______________________________________________
>>>>> users mailing list
>>>>> users_at_[hidden]
>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>> _______________________________________________
>>>> users mailing list
>>>> users_at_[hidden]
>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>
>>> _______________________________________________
>>> users mailing list
>>> users_at_[hidden]
>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>
>> _______________________________________________
>> users mailing list
>> users_at_[hidden]
>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>
>
> _______________________________________________
> users mailing list
> users_at_[hidden]
> http://www.open-mpi.org/mailman/listinfo.cgi/users

-- 
Jeff Squyres
jsquyres_at_[hidden]
For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/