Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: Re: [OMPI users] Binding to Core Warning
From: Saliya Ekanayake (esaliya_at_[hidden])
Date: 2014-02-27 00:33:41


Thank you. Anyway, your email contains good amount of info.

Saliya

On Wed, Feb 26, 2014 at 7:48 PM, Ralph Castain <rhc_at_[hidden]> wrote:

> I did one "chapter" of it on Jeff's blog and probably should complete it.
> Definitely need to update the FAQ for the new options.
>
> Sadly, outside of that and the mpirun man page, there isn't much available
> yet. I'm woefully far behind on it.
>
>
> On Feb 26, 2014, at 4:47 PM, Saliya Ekanayake <esaliya_at_[hidden]> wrote:
>
> Thank you Ralph, this is very insightful and I think I can better
> understand performance of our application.
>
> If I may ask, is there a document describing this affinity options? I've
> been looking at tuning FAQ and Jeff's blog posts.
>
> Thank you,
> Saliya
>
>
> On Wed, Feb 26, 2014 at 7:34 PM, Ralph Castain <rhc_at_[hidden]> wrote:
>
>>
>> On Feb 26, 2014, at 4:29 PM, Saliya Ekanayake <esaliya_at_[hidden]> wrote:
>>
>> I see, so if I understand correctly, the best scenario for threads would
>> be to bind 2 procs to sockets as --map-by socket:pe=4 and use 4 threads in
>> each proc.
>>
>>
>> Yes, that would be the best solution. If you have 4 cores in each socket,
>> then just bind each proc to the socket:
>>
>> --map-by socket --bind-to socket
>>
>> If you want to put one proc on each socket by itself, then do
>>
>> --map-by ppr:1:socket --bind-to socket
>>
>>
>>
>> Also, as you've mentioned binding threads to get memory locality, I guess
>> this has to be done at application level and not an option in OMPI
>>
>>
>> Sadly yes - the problem is that MPI lacks an init call for each thread,
>> and so we don't see the threads being started. You can use hwloc to bind
>> each thread, but it has to be done in the app itself.
>>
>>
>> Thank you,
>> Saliya
>>
>>
>> On Wed, Feb 26, 2014 at 4:50 PM, Ralph Castain <rhc_at_[hidden]> wrote:
>>
>>> Sorry, had to run some errands.
>>>
>>> On Feb 26, 2014, at 1:03 PM, Saliya Ekanayake <esaliya_at_[hidden]> wrote:
>>>
>>> Is it possible to bind to cores of multiple sockets? Say I have a
>>> machine with 2 sockets each with 4 cores and if I run 8 threads with 1 proc
>>> can I utilize all 8 cores for 8 threads?
>>>
>>>
>>> In that scenario, you won't get any benefit from binding as we only bind
>>> at the proc level (and binding to the entire node does nothing). You might
>>> want to bind your threads, however, as otherwise the threads will not
>>> necessarily execute local to any memory they malloc.
>>>
>>>
>>> Thank you for speedy replies
>>>
>>> Saliya
>>>
>>>
>>> On Wed, Feb 26, 2014 at 3:21 PM, Ralph Castain <rhc_at_[hidden]> wrote:
>>>
>>>>
>>>> On Feb 26, 2014, at 12:17 PM, Saliya Ekanayake <esaliya_at_[hidden]>
>>>> wrote:
>>>>
>>>> I have a followup question on this. In our application we have parallel
>>>> for loops similar to OMP parallel for. I noticed that in order to gain
>>>> speedup with threads I've to set --bind-to none, otherwise multiple threads
>>>> will bind to same core giving no increase in performance. For example, I
>>>> get following (attached) performance for a simple 3point stencil
>>>> computation run with T threads on 1 MPI process on 1 node (Tx1x1).
>>>>
>>>> My understanding is even when there are multiple procs per node we
>>>> should use --bind-to none in order to get performance with threads. Is this
>>>> correct? Also, what's the disadvantage of not using --bind-to core?
>>>>
>>>>
>>>> Your best performance with threads comes when you bind each process to
>>>> multiple cores. Binding helps performance by ensuring your memory is always
>>>> local, and provides some optimized scheduling benefits. You can bind to
>>>> multiple cores by adding the qualifier "pe=N" to your mapping definition,
>>>> like this:
>>>>
>>>> mpirun --map-by socket:pe=4 ....
>>>>
>>>> The above example will map processes by socket, and bind each process
>>>> to 4 cores.
>>>>
>>>> HTH
>>>> Ralph
>>>>
>>>>
>>>> Thank you,
>>>> Saliya
>>>>
>>>>
>>>> On Wed, Feb 26, 2014 at 11:01 AM, Saliya Ekanayake <esaliya_at_[hidden]>wrote:
>>>>
>>>>> Thank you Ralph, I'll check this.
>>>>>
>>>>>
>>>>> On Wed, Feb 26, 2014 at 10:04 AM, Ralph Castain <rhc_at_[hidden]>wrote:
>>>>>
>>>>>> It means that OMPI didn't get built against libnuma, and so we can't
>>>>>> ensure that memory is being bound local to the proc binding. Check to see
>>>>>> if numactl and numactl-devel are installed, or you can turn off the warning
>>>>>> using "-mca hwloc_base_mem_bind_failure_action silent"
>>>>>>
>>>>>>
>>>>>> On Feb 25, 2014, at 10:32 PM, Saliya Ekanayake <esaliya_at_[hidden]>
>>>>>> wrote:
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I tried to run an MPI Java program with --bind-to core. I receive the
>>>>>> following warning and wonder how to fix this.
>>>>>>
>>>>>>
>>>>>> WARNING: a request was made to bind a process. While the system
>>>>>> supports binding the process itself, at least one node does NOT
>>>>>> support binding memory to the process location.
>>>>>>
>>>>>> Node: 192.168.0.19
>>>>>>
>>>>>> This is a warning only; your job will continue, though performance may
>>>>>> be degraded.
>>>>>>
>>>>>>
>>>>>> Thank you,
>>>>>> Saliya
>>>>>>
>>>>>> --
>>>>>> Saliya Ekanayake esaliya_at_[hidden]
>>>>>> Cell 812-391-4914 Home 812-961-6383
>>>>>> http://saliya.org
>>>>>> _______________________________________________
>>>>>> users mailing list
>>>>>> users_at_[hidden]
>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>>>
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> users mailing list
>>>>>> users_at_[hidden]
>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Saliya Ekanayake esaliya_at_[hidden]
>>>>> Cell 812-391-4914 Home 812-961-6383
>>>>> http://saliya.org
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Saliya Ekanayake esaliya_at_[hidden]
>>>> Cell 812-391-4914 Home 812-961-6383
>>>> http://saliya.org
>>>> <3pointstencil.png>_______________________________________________
>>>>
>>>> users mailing list
>>>> users_at_[hidden]
>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> users mailing list
>>>> users_at_[hidden]
>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>
>>>
>>>
>>>
>>> --
>>> Saliya Ekanayake esaliya_at_[hidden]
>>> Cell 812-391-4914 Home 812-961-6383
>>> http://saliya.org
>>> _______________________________________________
>>> users mailing list
>>> users_at_[hidden]
>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>
>>>
>>>
>>> _______________________________________________
>>> users mailing list
>>> users_at_[hidden]
>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>
>>
>>
>>
>> --
>> Saliya Ekanayake esaliya_at_[hidden]
>> Cell 812-391-4914 Home 812-961-6383
>> http://saliya.org
>> _______________________________________________
>> users mailing list
>> users_at_[hidden]
>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>
>>
>>
>> _______________________________________________
>> users mailing list
>> users_at_[hidden]
>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>
>
>
>
> --
> Saliya Ekanayake esaliya_at_[hidden]
> Cell 812-391-4914 Home 812-961-6383
> http://saliya.org
> _______________________________________________
> users mailing list
> users_at_[hidden]
> http://www.open-mpi.org/mailman/listinfo.cgi/users
>
>
>
> _______________________________________________
> users mailing list
> users_at_[hidden]
> http://www.open-mpi.org/mailman/listinfo.cgi/users
>

-- 
Saliya Ekanayake esaliya_at_[hidden]
Cell 812-391-4914 Home 812-961-6383
http://saliya.org