Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: Re: [OMPI users] Binding to Core Warning
From: Saliya Ekanayake (esaliya_at_[hidden])
Date: 2014-02-26 19:47:14


Thank you Ralph, this is very insightful and I think I can better
understand performance of our application.

If I may ask, is there a document describing this affinity options? I've
been looking at tuning FAQ and Jeff's blog posts.

Thank you,
Saliya

On Wed, Feb 26, 2014 at 7:34 PM, Ralph Castain <rhc_at_[hidden]> wrote:

>
> On Feb 26, 2014, at 4:29 PM, Saliya Ekanayake <esaliya_at_[hidden]> wrote:
>
> I see, so if I understand correctly, the best scenario for threads would
> be to bind 2 procs to sockets as --map-by socket:pe=4 and use 4 threads in
> each proc.
>
>
> Yes, that would be the best solution. If you have 4 cores in each socket,
> then just bind each proc to the socket:
>
> --map-by socket --bind-to socket
>
> If you want to put one proc on each socket by itself, then do
>
> --map-by ppr:1:socket --bind-to socket
>
>
>
> Also, as you've mentioned binding threads to get memory locality, I guess
> this has to be done at application level and not an option in OMPI
>
>
> Sadly yes - the problem is that MPI lacks an init call for each thread,
> and so we don't see the threads being started. You can use hwloc to bind
> each thread, but it has to be done in the app itself.
>
>
> Thank you,
> Saliya
>
>
> On Wed, Feb 26, 2014 at 4:50 PM, Ralph Castain <rhc_at_[hidden]> wrote:
>
>> Sorry, had to run some errands.
>>
>> On Feb 26, 2014, at 1:03 PM, Saliya Ekanayake <esaliya_at_[hidden]> wrote:
>>
>> Is it possible to bind to cores of multiple sockets? Say I have a machine
>> with 2 sockets each with 4 cores and if I run 8 threads with 1 proc can I
>> utilize all 8 cores for 8 threads?
>>
>>
>> In that scenario, you won't get any benefit from binding as we only bind
>> at the proc level (and binding to the entire node does nothing). You might
>> want to bind your threads, however, as otherwise the threads will not
>> necessarily execute local to any memory they malloc.
>>
>>
>> Thank you for speedy replies
>>
>> Saliya
>>
>>
>> On Wed, Feb 26, 2014 at 3:21 PM, Ralph Castain <rhc_at_[hidden]> wrote:
>>
>>>
>>> On Feb 26, 2014, at 12:17 PM, Saliya Ekanayake <esaliya_at_[hidden]>
>>> wrote:
>>>
>>> I have a followup question on this. In our application we have parallel
>>> for loops similar to OMP parallel for. I noticed that in order to gain
>>> speedup with threads I've to set --bind-to none, otherwise multiple threads
>>> will bind to same core giving no increase in performance. For example, I
>>> get following (attached) performance for a simple 3point stencil
>>> computation run with T threads on 1 MPI process on 1 node (Tx1x1).
>>>
>>> My understanding is even when there are multiple procs per node we
>>> should use --bind-to none in order to get performance with threads. Is this
>>> correct? Also, what's the disadvantage of not using --bind-to core?
>>>
>>>
>>> Your best performance with threads comes when you bind each process to
>>> multiple cores. Binding helps performance by ensuring your memory is always
>>> local, and provides some optimized scheduling benefits. You can bind to
>>> multiple cores by adding the qualifier "pe=N" to your mapping definition,
>>> like this:
>>>
>>> mpirun --map-by socket:pe=4 ....
>>>
>>> The above example will map processes by socket, and bind each process to
>>> 4 cores.
>>>
>>> HTH
>>> Ralph
>>>
>>>
>>> Thank you,
>>> Saliya
>>>
>>>
>>> On Wed, Feb 26, 2014 at 11:01 AM, Saliya Ekanayake <esaliya_at_[hidden]>wrote:
>>>
>>>> Thank you Ralph, I'll check this.
>>>>
>>>>
>>>> On Wed, Feb 26, 2014 at 10:04 AM, Ralph Castain <rhc_at_[hidden]>wrote:
>>>>
>>>>> It means that OMPI didn't get built against libnuma, and so we can't
>>>>> ensure that memory is being bound local to the proc binding. Check to see
>>>>> if numactl and numactl-devel are installed, or you can turn off the warning
>>>>> using "-mca hwloc_base_mem_bind_failure_action silent"
>>>>>
>>>>>
>>>>> On Feb 25, 2014, at 10:32 PM, Saliya Ekanayake <esaliya_at_[hidden]>
>>>>> wrote:
>>>>>
>>>>> Hi,
>>>>>
>>>>> I tried to run an MPI Java program with --bind-to core. I receive the
>>>>> following warning and wonder how to fix this.
>>>>>
>>>>>
>>>>> WARNING: a request was made to bind a process. While the system
>>>>> supports binding the process itself, at least one node does NOT
>>>>> support binding memory to the process location.
>>>>>
>>>>> Node: 192.168.0.19
>>>>>
>>>>> This is a warning only; your job will continue, though performance may
>>>>> be degraded.
>>>>>
>>>>>
>>>>> Thank you,
>>>>> Saliya
>>>>>
>>>>> --
>>>>> Saliya Ekanayake esaliya_at_[hidden]
>>>>> Cell 812-391-4914 Home 812-961-6383
>>>>> http://saliya.org
>>>>> _______________________________________________
>>>>> users mailing list
>>>>> users_at_[hidden]
>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> users mailing list
>>>>> users_at_[hidden]
>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Saliya Ekanayake esaliya_at_[hidden]
>>>> Cell 812-391-4914 Home 812-961-6383
>>>> http://saliya.org
>>>>
>>>
>>>
>>>
>>> --
>>> Saliya Ekanayake esaliya_at_[hidden]
>>> Cell 812-391-4914 Home 812-961-6383
>>> http://saliya.org
>>> <3pointstencil.png>_______________________________________________
>>>
>>> users mailing list
>>> users_at_[hidden]
>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>
>>>
>>>
>>> _______________________________________________
>>> users mailing list
>>> users_at_[hidden]
>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>
>>
>>
>>
>> --
>> Saliya Ekanayake esaliya_at_[hidden]
>> Cell 812-391-4914 Home 812-961-6383
>> http://saliya.org
>> _______________________________________________
>> users mailing list
>> users_at_[hidden]
>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>
>>
>>
>> _______________________________________________
>> users mailing list
>> users_at_[hidden]
>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>
>
>
>
> --
> Saliya Ekanayake esaliya_at_[hidden]
> Cell 812-391-4914 Home 812-961-6383
> http://saliya.org
> _______________________________________________
> users mailing list
> users_at_[hidden]
> http://www.open-mpi.org/mailman/listinfo.cgi/users
>
>
>
> _______________________________________________
> users mailing list
> users_at_[hidden]
> http://www.open-mpi.org/mailman/listinfo.cgi/users
>

-- 
Saliya Ekanayake esaliya_at_[hidden]
Cell 812-391-4914 Home 812-961-6383
http://saliya.org