Hi Jean-François ;)
Do you have the same behavior if you disable openib at run time ? :

--mca btl ^openib

My experience with the OpenIB BTL is that its inner threading is bugged.

Maxime

Le 2013-11-28 19:21, Jean-Francois St-Pierre a écrit :
Hi,
I've compiled ompi1.6.5 with multi-thread support (using Intel
compilers 12.0.4.191, but I get the same result with gcc) :

./configure --with-tm/opt/torque --with-openib
--enable-mpi-thread-multiple CC=icc CXX=icpc F77=ifort FC=ifort

And i've built a simple sample code that only does the Init and one
MPI_Barrier. The core of the code is :

  setbuf(stdout, NULL);
  MPI_Init_thread(&argc, &argv, MPI_THREAD_MULTIPLE, &provided);
  fprintf(stdout,"%6d: Provided thread support %d ", getpid(), provided);
  int flag,claimed;
  MPI_Is_thread_main( &flag );
  MPI_Query_thread( &claimed );

  fprintf(stdout,"%6d: Before Comm_rank, flag %d, claimed %d \n",
getpid(), flag, claimed);
  MPI_Comm_rank(MPI_COMM_WORLD, &gRank);

  fprintf(stdout,"%6d: Comm_size rank %d\n",getpid(), gRank);
  MPI_Comm_size(MPI_COMM_WORLD, &gNTasks);

  fprintf(stdout,"%6d: Before Barrier\n", getpid());
  MPI_Barrier( MPI_COMM_WORLD );

  fprintf(stdout,"%6d: After Barrier\n", getpid());
  MPI_Finalize();

When I launch it on 2 nodes (mono-core per node) I get this sample output :

/***  Output
 mpirun -pernode -np 2 sample_code
 7356: Provided thread support 3 MPI_THREAD_MULTIPLE
 7356: Before Comm_rank, flag 1, claimed 3
 7356: Comm_size rank 0
 7356: Before Barrier
 26277: Provided thread support 3 MPI_THREAD_MULTIPLE
 26277: Before Comm_rank, flag 1, claimed 3
 26277: Comm_size rank 1
 26277: Before Barrier
 ^Cmpirun: killing job...
 */

The deadlock never gets over the MPI_Barrier when I use
MPI_THREAD_MULTIPLE, but it runs fine using MPI_THREAD_SERIALIZED .  I
get the same behavior with ompi 1.7.3. I don't get a deadlock when the
2 MPI processes are hosted on the same node.

In attachement, you'll find my config.out, config.log, environment
variables on the execution node, both make.out, sample code and output
etc.

Thanks,

Jeff


_______________________________________________
users mailing list
users@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users


-- 
---------------------------------
Maxime Boissonneault
Analyste de calcul - Calcul Québec, Université Laval
Ph. D. en physique