Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: Re: [OMPI users] pinning processes by default
From: Albert Solernou (albert.solernou_at_[hidden])
Date: 2014-05-23 11:15:38


Very useful!
Thank you Ralph.

Albert

On Fri 23 May 2014 15:26:58 BST, Ralph Castain wrote:
>
> On May 23, 2014, at 7:14 AM, Albert Solernou <albert.solernou_at_[hidden]> wrote:
>
>> Well,
>> the problem is that I don't know how to do any of these things
>
> Ah! You might want to read this:
>
> http://www.open-mpi.org/faq/?category=tuning#mca-params
>
>> , so more explicitly:
>> - does OpenMPI accept any environment variable that prevents binding like MV2_ENABLE_AFFINITY does on MVAPICH2?
>
> OMPI_MCA_hwloc_base_binding_policy=none
>
>> - what is the default mca param file? Is it a runtime file? or a configuration file? how do I modify it to achieve that?
>
> When you install OMPI, an "etc" directory gets created under the prefix location. In that directory is a file "openmpi-mca-params.conf". This is your default MCA param file that mpirun (and every OMPI process) reads on startup. You can put any params in there that you want. In this case, you'd add a line:
>
> hwloc_base_binding_policy = none
>
> HTH
> Ralph
>
>>
>> Thanks,
>> Albert
>>
>> On 23/05/14 15:02, Ralph Castain wrote:
>>>
>>> On May 23, 2014, at 6:58 AM, Albert Solernou <albert.solernou_at_[hidden]> wrote:
>>>
>>>> Hi,
>>>> thanks a lot for your quick answers, and I see my error, it is "--bind-to none" instead of "--bind-to-none".
>>>>
>>>> However, I need to be able to run "mpirun -np 2" without any binding argument and get a "--bind-to none" behaviour. I don't know if I can export an environment variable to do that, and I don't mind to re-compile with some flag I missed or to alter the code.
>>>>
>>>> Any suggestion?
>>>
>>> Obviously, you could set an variable in your environment, so I'm assuming there is some other limitation in play here? If so, you could always put the MCA param in the default mca param file - we'll pick it up from there.
>>>
>>>>
>>>> Albert
>>>>
>>>> On 23/05/14 14:32, Ralph Castain wrote:
>>>>> Note that the lama mapper described in those slides may not work as it hasn't been maintained in a while. However, you can use the map-by and bind-to options to do the same things.
>>>>>
>>>>> If you want to disable binding, you can do so by adding "--bind-to none" to the cmd line, or via the MCA param "hwloc_base_binding_policy=none"
>>>>>
>>>>> If you want to bind your process to multiple cores (say one per thread), then you can use "--map-by core:pe=N". Many hybrid users prefer to bind to a socket, with one process for each socket - that can be done with "--map-by socket --bind-to socket". This keeps all the threads in the same NUMA domain. If you aren't sure that each socket is its own NUMA domain, you can alternatively "--map-by numa --bind-to numa" - that'll keep you in your own NUMA domain regardless of whether that's at the socket level or elsewhere.
>>>>>
>>>>> I'm working on adding a full description of the mapping/binding system to our web site.
>>>>>
>>>>> HTH
>>>>> Ralph
>>>>>
>>>>> On May 23, 2014, at 6:22 AM, Brock Palen <brockp_at_[hidden]> wrote:
>>>>>
>>>>>> Albert,
>>>>>>
>>>>>> Actually doing affinity correctly for hybrid got easier in OpenMPI 1.7 and newer, In the past you had to make a lot of assumptions, stride by node etc,
>>>>>>
>>>>>> Now you can define a layout:
>>>>>>
>>>>>> http://blogs.cisco.com/performance/eurompi13-cisco-slides-open-mpi-process-affinity-user-interface/
>>>>>>
>>>>>> Brock Palen
>>>>>> www.umich.edu/~brockp
>>>>>> CAEN Advanced Computing
>>>>>> XSEDE Campus Champion
>>>>>> brockp_at_[hidden]
>>>>>> (734)936-1985
>>>>>>
>>>>>>
>>>>>>
>>>>>> On May 23, 2014, at 9:19 AM, Albert Solernou <albert.solernou_at_[hidden]> wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>> after compiling and installing OpenMPI 1.8.1, I find that OpenMPI is pinning processes onto cores. Although this may be
>>>>>>> desirable on some cases, it is a complete disaster when runnning hybrid OpenMP-MPI applications. Therefore, I want to disable this behaviour, but don't know how.
>>>>>>>
>>>>>>> I configured OpenMPI with:
>>>>>>> ./configure \
>>>>>>> --prefix=$OPENMPI_HOME \
>>>>>>> --with-psm \
>>>>>>> --with-tm=/system/software/arcus/lib/PBS/11.3.0.121723 \
>>>>>>> --enable-mpirun-prefix-by-default \
>>>>>>> --enable-mpi-thread-multiple
>>>>>>>
>>>>>>> and:
>>>>>>> ompi_info | grep paffinity
>>>>>>> does not report anything. However,
>>>>>>> mpirun -np 2 --report-bindings ./wrk
>>>>>>> reports bindings:
>>>>>>> [login3:04574] MCW rank 1 bound to socket 0[core 1[hwt 0-1]]: [../BB/../../../../../..][../../../../../../../..]
>>>>>>> [login3:04574] MCW rank 0 bound to socket 0[core 0[hwt 0-1]]: [BB/../../../../../../..][../../../../../../../..]
>>>>>>> but they cannot be disabled as:
>>>>>>> mpirun -np 2 --bind-to-none ./wrk
>>>>>>> returns:
>>>>>>> mpirun: Error: unknown option "--bind-to-none"
>>>>>>>
>>>>>>> Any idea on what went wrong?
>>>>>>>
>>>>>>> Best,
>>>>>>> Albert
>>>>>>>
>>>>>>> --
>>>>>>> ---------------------------------
>>>>>>> Dr. Albert Solernou
>>>>>>> Research Associate
>>>>>>> Oxford Supercomputing Centre,
>>>>>>> University of Oxford
>>>>>>> Tel: +44 (0)1865 610631
>>>>>>> ---------------------------------
>>>>>>> _______________________________________________
>>>>>>> users mailing list
>>>>>>> users_at_[hidden]
>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>>>
>>>>>> _______________________________________________
>>>>>> users mailing list
>>>>>> users_at_[hidden]
>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>>
>>>>> _______________________________________________
>>>>> users mailing list
>>>>> users_at_[hidden]
>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>>
>>>>
>>>> --
>>>> ---------------------------------
>>>> Dr. Albert Solernou
>>>> Research Associate
>>>> Oxford Supercomputing Centre,
>>>> University of Oxford
>>>> Tel: +44 (0)1865 610631
>>>> ---------------------------------
>>>> _______________________________________________
>>>> users mailing list
>>>> users_at_[hidden]
>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>
>>> _______________________________________________
>>> users mailing list
>>> users_at_[hidden]
>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>
>>
>> --
>> ---------------------------------
>> Dr. Albert Solernou
>> Research Associate
>> Oxford Supercomputing Centre,
>> University of Oxford
>> Tel: +44 (0)1865 610631
>> ---------------------------------
>> _______________________________________________
>> users mailing list
>> users_at_[hidden]
>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>
> _______________________________________________
> users mailing list
> users_at_[hidden]
> http://www.open-mpi.org/mailman/listinfo.cgi/users

--
---------------------------------
  Dr. Albert Solernou
  Research Associate
  Oxford Supercomputing Centre,
  University of Oxford
  Tel: +44 (0)1865 610631
---------------------------------