Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: Re: [OMPI users] MPI_THREAD_MULTIPLE causes deadlock in simple MPI_Barrier case (ompi 1.6.5 and 1.7.3)
From: Jean-Francois St-Pierre (jf.stpierre_at_[hidden])
Date: 2013-11-29 12:21:11


Hi Maxim,

I didn't get any change of behavior using your parameter set, neither
with --mca btl tcp,sm,self. It is still dead locked at the barrier.

--
Jean-François St-Pierre, analyste en calcul scientifique
jf.stpierre_at_[hidden]
bureau S-250, pavillon Roger-Gaudry (principal), Université de Montréal
téléphone : 514 343-6111 poste 10024     télécopieur : 514 343-2155
Calcul Québec (www.calculquebec.ca)
Calcul Canada (calculcanada.ca)
On Fri, Nov 29, 2013 at 10:00 AM, Maxime Boissonneault
<maxime.boissonneault_at_[hidden]> wrote:
> Hi Jean-François ;)
> Do you have the same behavior if you disable openib at run time ? :
>
> --mca btl ^openib
>
> My experience with the OpenIB BTL is that its inner threading is bugged.
>
> Maxime
>
> Le 2013-11-28 19:21, Jean-Francois St-Pierre a écrit :
>
> Hi,
> I've compiled ompi1.6.5 with multi-thread support (using Intel
> compilers 12.0.4.191, but I get the same result with gcc) :
>
> ./configure --with-tm/opt/torque --with-openib
> --enable-mpi-thread-multiple CC=icc CXX=icpc F77=ifort FC=ifort
>
> And i've built a simple sample code that only does the Init and one
> MPI_Barrier. The core of the code is :
>
>   setbuf(stdout, NULL);
>   MPI_Init_thread(&argc, &argv, MPI_THREAD_MULTIPLE, &provided);
>   fprintf(stdout,"%6d: Provided thread support %d ", getpid(), provided);
>   int flag,claimed;
>   MPI_Is_thread_main( &flag );
>   MPI_Query_thread( &claimed );
>
>   fprintf(stdout,"%6d: Before Comm_rank, flag %d, claimed %d \n",
> getpid(), flag, claimed);
>   MPI_Comm_rank(MPI_COMM_WORLD, &gRank);
>
>   fprintf(stdout,"%6d: Comm_size rank %d\n",getpid(), gRank);
>   MPI_Comm_size(MPI_COMM_WORLD, &gNTasks);
>
>   fprintf(stdout,"%6d: Before Barrier\n", getpid());
>   MPI_Barrier( MPI_COMM_WORLD );
>
>   fprintf(stdout,"%6d: After Barrier\n", getpid());
>   MPI_Finalize();
>
> When I launch it on 2 nodes (mono-core per node) I get this sample output :
>
> /***  Output
>  mpirun -pernode -np 2 sample_code
>  7356: Provided thread support 3 MPI_THREAD_MULTIPLE
>  7356: Before Comm_rank, flag 1, claimed 3
>  7356: Comm_size rank 0
>  7356: Before Barrier
>  26277: Provided thread support 3 MPI_THREAD_MULTIPLE
>  26277: Before Comm_rank, flag 1, claimed 3
>  26277: Comm_size rank 1
>  26277: Before Barrier
>  ^Cmpirun: killing job...
>  */
>
> The deadlock never gets over the MPI_Barrier when I use
> MPI_THREAD_MULTIPLE, but it runs fine using MPI_THREAD_SERIALIZED .  I
> get the same behavior with ompi 1.7.3. I don't get a deadlock when the
> 2 MPI processes are hosted on the same node.
>
> In attachement, you'll find my config.out, config.log, environment
> variables on the execution node, both make.out, sample code and output
> etc.
>
> Thanks,
>
> Jeff
>
>
>
> _______________________________________________
> users mailing list
> users_at_[hidden]
> http://www.open-mpi.org/mailman/listinfo.cgi/users
>
>
>
> --
> ---------------------------------
> Maxime Boissonneault
> Analyste de calcul - Calcul Québec, Université Laval
> Ph. D. en physique
>
>
> _______________________________________________
> users mailing list
> users_at_[hidden]
> http://www.open-mpi.org/mailman/listinfo.cgi/users