Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: Re: [OMPI users] Hybrid OpenMPI / OpenMP run pins OpenMP threads to a single core
From: David Akin (nospamou_at_[hidden])
Date: 2010-07-29 01:11:09


Below are all places that could contain mca related settings.

grep -i mca /usr/mpi/gcc/openmpi-1.4-qlc/etc/openmpi-mca-params.conf
# This is the default system-wide MCA parameters defaults file.
# Specifically, the MCA parameter "mca_param_files" defaults to a
# "$HOME/.openmpi/mca-params.conf:$sysconf/openmpi-mca-params.conf"
# mca_param_files is not changed, this file is used to set system-wide
# MCA parameters. This file can therefore be used to set system-wide
# default MCA parameters for all users. Of course, users can override
# for setting system-specific MCA parameters for those users who don't
# during their startup to determine what default values for MCA
# The format is straightforward: one per line, mca_param_name =
# See "ompi_info --param all all" for a full listing of Open MPI MCA

ls -l $HOME/.openmpi/mca-params.conf
ls: /home/dave/.openmpi/mca-params.conf: No such file or directory

/usr/mpi/gcc/openmpi-1.4-qlc/bin/mpirun -host c005 -np 1 env|grep -i mca
OMPI_MCA_orte_precondition_transports=db26e173df59af40-3609f2dcfea5746b
OMPI_MCA_orte_local_daemon_uri=999882752.0;tcp://10.1.5.5:60361
OMPI_MCA_orte_hnp_uri=999882752.0;tcp://10.1.5.5:60361
OMPI_MCA_mpi_yield_when_idle=0
OMPI_MCA_orte_app_num=0
OMPI_MCA_ess=env
OMPI_MCA_orte_ess_num_procs=1
OMPI_MCA_orte_ess_jobid=999882753
OMPI_MCA_orte_ess_vpid=0

On Wed, Jul 28, 2010 at 10:50 PM, Ralph Castain <rhc_at_[hidden]> wrote:
> Something doesn't add up - the default for ompi is to -not- bind. Check your default mca param file and your environment. Do you have any mca params set in them?
>
>
> On Jul 28, 2010, at 9:40 PM, David Akin wrote:
>
>> Here's the exact command I'm running when all threads *are* pinned to
>> a single core:
>>
>> /usr/mpi/gcc/openmpi-1.4-qlc/bin/mpirun -host c005,c006 -np 2
>> OMP_NUM_THREADS=4 hybrid4.gcc
>>
>> Can anyone verify they have the same issue?
>>
>> On Wed, Jul 28, 2010 at 7:52 PM, Ralph Castain <rhc_at_[hidden]> wrote:
>>> How are you running it when the threads are all on one core?
>>>
>>> If you are specifying --bind-to-core, then of course all the threads will be on one core since we bind the process (not the thread). If you are specifying -mca mpi_paffinity_alone 1, then the same behavior results.
>>>
>>> Generally, if you want to bind threads, the only way to do it is with a rank file. We -might- figure out a way to provide an interface for thread-level binding, but I'm not sure about that right now. As things stand, OMPI has no visibility into the fact that your app spawned threads.
>>>
>>>
>>> On Jul 28, 2010, at 5:47 PM, David Akin wrote:
>>>
>>>> All,
>>>> I'm trying to get the OpenMP portion of the code below to run
>>>> multicore on a couple of 8 core nodes.
>>>>
>>>> Good news: multiple threads are being spawned on each node in the run.
>>>> Bad news: each of the threads only runs on a single core, leaving 7
>>>> cores basically idle.
>>>> Sorta good news: if I provide a rank file I get the threads running on
>>>> different cores within each node (PITA.
>>>>
>>>> Here's the first lines of output.
>>>>
>>>> /usr/mpi/gcc/openmpi-1.4-qlc/bin/mpirun -host c005,c006 -np 2 -rf
>>>> rank.file -x OMP_NUM_THREADS=4 hybrid4.gcc
>>>>
>>>> Hello from thread 2 out of 4 from process 1 out of 2 on c006.local
>>>> another parallel region:       name:c006.local MPI_RANK_ID=1 OMP_THREAD_ID=2
>>>> Hello from thread 3 out of 4 from process 1 out of 2 on c006.local
>>>> another parallel region:       name:c006.local MPI_RANK_ID=1 OMP_THREAD_ID=3
>>>> Hello from thread 1 out of 4 from process 1 out of 2 on c006.local
>>>> another parallel region:       name:c006.local MPI_RANK_ID=1 OMP_THREAD_ID=1
>>>> Hello from thread 1 out of 4 from process 0 out of 2 on c005.local
>>>> another parallel region:       name:c005.local MPI_RANK_ID=0 OMP_THREAD_ID=1
>>>> Hello from thread 3 out of 4 from process 0 out of 2 on c005.local
>>>> Hello from thread 2 out of 4 from process 0 out of 2 on c005.local
>>>> another parallel region:       name:c005.local MPI_RANK_ID=0 OMP_THREAD_ID=3
>>>> another parallel region:       name:c005.local MPI_RANK_ID=0 OMP_THREAD_ID=2
>>>> Hello from thread 0 out of 4 from process 0 out of 2 on c005.local
>>>> another parallel region:       name:c005.local MPI_RANK_ID=0 OMP_THREAD_ID=0
>>>> Hello from thread 0 out of 4 from process 1 out of 2 on c006.local
>>>> another parallel region:       name:c006.local MPI_RANK_ID=1 OMP_THREAD_ID=0
>>>> another parallel region:       name:c005.local MPI_RANK_ID=0 OMP_THREAD_ID=3
>>>> another parallel region:       name:c005.local MPI_RANK_ID=0 OMP_THREAD_ID=2
>>>> another parallel region:       name:c005.local MPI_RANK_ID=0 OMP_THREAD_ID=0
>>>> another parallel region:       name:c006.local MPI_RANK_ID=1 OMP_THREAD_ID=3
>>>> another parallel region:       name:c005.local MPI_RANK_ID=0 OMP_THREAD_ID=3
>>>> another parallel region:       name:c005.local MPI_RANK_ID=0 OMP_THREAD_ID=2
>>>> another parallel region:       name:c006.local MPI_RANK_ID=1 OMP_THREAD_ID=0
>>>> another parallel region:       name:c006.local MPI_RANK_ID=1 OMP_THREAD_ID=1
>>>> .
>>>> .
>>>> .
>>>>
>>>> Here's the simple code:
>>>> #include <stdio.h>
>>>> #include "mpi.h"
>>>> #include <omp.h>
>>>>
>>>> int main(int argc, char *argv[]) {
>>>>  int numprocs, rank, namelen;
>>>>  char processor_name[MPI_MAX_PROCESSOR_NAME];
>>>>  int iam = 0, np = 1;
>>>>  char name[MPI_MAX_PROCESSOR_NAME];   /* MPI_MAX_PROCESSOR_NAME ==
>>>> 128         */
>>>>  int O_ID;                            /* OpenMP thread ID
>>>>         */
>>>>  int M_ID;                            /* MPI rank ID
>>>>         */
>>>>  int rtn_val;
>>>>
>>>>  MPI_Init(&argc, &argv);
>>>>  MPI_Comm_size(MPI_COMM_WORLD, &numprocs);
>>>>  MPI_Comm_rank(MPI_COMM_WORLD, &rank);
>>>>  MPI_Get_processor_name(processor_name, &namelen);
>>>>
>>>>  #pragma omp parallel default(shared) private(iam, np,O_ID)
>>>>  {
>>>>    np = omp_get_num_threads();
>>>>    iam = omp_get_thread_num();
>>>>    printf("Hello from thread %d out of %d from process %d out of %d on %s\n",
>>>>           iam, np, rank, numprocs, processor_name);
>>>>    int i=0;
>>>>    int j=0;
>>>>    double counter=0;
>>>>    for(i =0;i<99999999;i++)
>>>>            {
>>>>             O_ID = omp_get_thread_num();          /* get OpenMP
>>>> thread ID                 */
>>>>             MPI_Get_processor_name(name,&namelen);
>>>>             rtn_val = MPI_Comm_rank(MPI_COMM_WORLD,&M_ID);
>>>>             printf("another parallel region:       name:%s
>>>> MPI_RANK_ID=%d OMP_THREAD_ID=%d\n", name,M_ID,O_ID);
>>>>             for(j = 0;j<999999999;j++)
>>>>              {
>>>>               counter=counter+i;
>>>>              }
>>>>            }
>>>>
>>>>  }
>>>>
>>>>  MPI_Finalize();
>>>>
>>>> }
>>>> _______________________________________________
>>>> users mailing list
>>>> users_at_[hidden]
>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>
>>>
>>> _______________________________________________
>>> users mailing list
>>> users_at_[hidden]
>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>
>>
>> _______________________________________________
>> users mailing list
>> users_at_[hidden]
>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>
>
> _______________________________________________
> users mailing list
> users_at_[hidden]
> http://www.open-mpi.org/mailman/listinfo.cgi/users
>