Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: [OMPI users] OpenMPI 1.5.4 remote send hang on Windows 2008R2
From: Randy Abernethy (randy.abernethy_at_[hidden])
Date: 2012-01-10 20:02:51


Hello,

I have run into an issue that appears to be related to sending messages to
multiple processes on a single remote host prior to the remote processes
sending messages to the origin. I have cooked the issue down to the
following:

*Test Environment of 3 Identical Hosts:*

· * Intel i7-2600K, 12GB ram, Intel GB Ethernet, DLink Switch

 · * Windows 2008R2 x64 with all current updates

· * OMPI (all three hosts report the same ompi_info and were
installed with the same binary)
http://www.open-mpi.org/software/ompi/v1.5/downloads/OpenMPI_v1.5.4-1_win64.exe

C:\GDX>ompi_info -v ompi full --parsable

package:Open MPI hpcfan_at_VISCLUSTER26 Distribution

ompi:version:full:1.5.4

ompi:version:svn:r25060

ompi:version:release_date:Aug 18, 2011

orte:version:full:1.5.4

orte:version:svn:r25060

orte:version:release_date:Aug 18, 2011

opal:version:full:1.5.4

opal:version:svn:r25060

opal:version:release_date:Aug 18, 2011

ident:1.5.4

*Test Program:*

#include <stdio.h>

#define OMPI_IMPORTS

#include "C:\Program Files (x86)\OpenMPI_v1.5.4-x64\include\mpi.h"

int main(int argc, char *argv[])

{

   int rank, size, i, msg;

   MPI_Init(&argc, &argv);

   MPI_Comm_rank(MPI_COMM_WORLD, &rank);

   MPI_Comm_size(MPI_COMM_WORLD, &size);

   printf("Process %i of %i initialized\n", rank, size);

   if (0 == rank) {

      for (i = 1; i < size; i++) {

         printf("Process %i sending %i to %i\n", rank, i, i);

         MPI_Send(&rank, 1, MPI_INT, i, 0, MPI_COMM_WORLD);

      }

      for (i = 1; i < size; i++) {

         MPI_Recv(&msg, 1, MPI_INT, MPI_ANY_SOURCE, MPI_ANY_TAG,
MPI_COMM_WORLD, MPI_STATUS_IGNORE);

         printf("Process %i received %i\n", rank, msg);

      }

   }

   else {

      MPI_Recv(&msg, 1, MPI_INT, MPI_ANY_SOURCE, MPI_ANY_TAG,
MPI_COMM_WORLD, MPI_STATUS_IGNORE);

      printf("Process %i received %i\n", rank, msg);

      MPI_Send(&rank, 1, MPI_INT, 0, 0, MPI_COMM_WORLD);

      printf("Process %i sent %i to %i\n", rank, rank, 0);

   }

   printf("Process %i exiting\n", rank);

   MPI_Finalize();

   return 0;

}

*Test Cases:*

· X procs on the originating node: Working

· X procs on the originating node and one proc on one or more
remote nodes: Working

· X procs on the originating node and more than one proc on any
remote node: Fails

A test with two procs on the origin and one on each of two remote nodes
runs, however the same test with the two remote procs on the same machine
fails on the second remote send. Here are some test runs (the ^C indicates
a hang).

C:\GDX>mpirun -v -display-map -hostfile mpihosts -np 2 c:\gdx\distmsg.exe

 ======================== JOB MAP ========================

 Data for node: Yap Num procs: 2

        Process OMPI jobid: [42094,1] Process rank: 0

        Process OMPI jobid: [42094,1] Process rank: 1

 =============================================================

Process 0 of 2 initialized

Process 1 of 2 initialized

Process 0 sending 1 to 1

Process 1 received 0

Process 1 sent 1 to 0

Process 1 exiting

Process 0 received 1

Process 0 exiting

C:\GDX>mpirun -v -display-map -hostfile mpihosts -np 3 c:\gdx\distmsg.exe

 ======================== JOB MAP ========================

 Data for node: Yap Num procs: 2

        Process OMPI jobid: [42014,1] Process rank: 0

        Process OMPI jobid: [42014,1] Process rank: 1

 Data for node: chuuk Num procs: 1

        Process OMPI jobid: [42014,1] Process rank: 2

 =============================================================

connecting to chuuk

username:administrator

password:********

Save Credential?(Y/N) n

Process 0 of 3 initialized

Process 1 of 3 initialized

Process 0 sending 1 to 1

Process 0 sending 2 to 2

Process 1 received 0

Process 1 sent 1 to 0

Process 1 exiting

Process 0 received 1

Process 0 received 2

Process 0 exiting

C:\GDX>mpirun -v -display-map -hostfile mpihosts -np 4 c:\gdx\distmsg.exe

 ======================== JOB MAP ========================

 Data for node: Yap Num procs: 2

        Process OMPI jobid: [43894,1] Process rank: 0

        Process OMPI jobid: [43894,1] Process rank: 1

 Data for node: chuuk Num procs: 2

        Process OMPI jobid: [43894,1] Process rank: 2

        Process OMPI jobid: [43894,1] Process rank: 3

 =============================================================

connecting to chuuk

username:administrator

password:********

Save Credential?(Y/N) n

Process 0 of 4 initialized

Process 1 of 4 initialized

Process 0 sending 1 to 1

Process 0 sending 2 to 2

Process 1 received 0

Process 1 sent 1 to 0

Process 1 exiting

Process 0 sending 3 to 3

^C

C:\GDX>mpirun -v -display-map -hostfile mpihosts -np 4 c:\gdx\distmsg.exe

 ======================== JOB MAP ========================

 Data for node: Yap Num procs: 2

        Process OMPI jobid: [43310,1] Process rank: 0

        Process OMPI jobid: [43310,1] Process rank: 1

 Data for node: chuuk Num procs: 1

        Process OMPI jobid: [43310,1] Process rank: 2

 Data for node: kosrae Num procs: 1

        Process OMPI jobid: [43310,1] Process rank: 3

 =============================================================

connecting to chuuk

username:administrator

password:********

Save Credential?(Y/N) n

connecting to kosrae

username:administrator

password:********

Save Credential?(Y/N) n

Process 0 of 4 initialized

Process 1 of 4 initialized

Process 0 sending 1 to 1

Process 0 sending 2 to 2

Process 1 received 0

Process 1 sent 1 to 0

Process 1 exiting

Process 0 sending 3 to 3

Process 0 received 1

Process 0 received 2

Process 0 received 3

Process 0 exiting

C:\GDX>mpirun -v -display-map -hostfile mpihosts -np 5 c:\gdx\distmsg.exe

 ======================== JOB MAP ========================

 Data for node: Yap Num procs: 2

        Process OMPI jobid: [43590,1] Process rank: 0

        Process OMPI jobid: [43590,1] Process rank: 1

 Data for node: chuuk Num procs: 2

        Process OMPI jobid: [43590,1] Process rank: 2

        Process OMPI jobid: [43590,1] Process rank: 3

 Data for node: kosrae Num procs: 1

        Process OMPI jobid: [43590,1] Process rank: 4

 =============================================================

connecting to chuuk

username:administrator

password:********

Save Credential?(Y/N) n

connecting to kosrae

username:administrator

password:********

Save Credential?(Y/N) n

Process 0 of 5 initialized

Process 1 of 5 initialized

Process 0 sending 1 to 1

Process 0 sending 2 to 2

Process 1 received 0

Process 1 sent 1 to 0

Process 1 exiting

Process 0 sending 3 to 3

^C

The remote process which is the target of the hung send seems to generate
significant ongoing CPU activity and "Other" I/O.

*Workaround*

Curiously swapping the send/receive order solves the problem.

#include <stdio.h>

#define OMPI_IMPORTS

#include "C:\Program Files (x86)\OpenMPI_v1.5.4-x64\include\mpi.h"

int main(int argc, char *argv[])

{

   int rank, size, i, msg;

   MPI_Init(&argc, &argv);

   MPI_Comm_rank(MPI_COMM_WORLD, &rank);

   MPI_Comm_size(MPI_COMM_WORLD, &size);

   printf("Process %i of %i initialized\n", rank, size);

   if (0 == rank) {

      for (i = 1; i < size; i++) {

         MPI_Recv(&msg, 1, MPI_INT, MPI_ANY_SOURCE, MPI_ANY_TAG,
MPI_COMM_WORLD, MPI_STATUS_IGNORE);

         printf("Process %i received %i\n", rank, msg);

      }

      for (i = 1; i < size; i++) {

         printf("Process %i sending %i to %i\n", rank, i, i);

         MPI_Send(&rank, 1, MPI_INT, i, 0, MPI_COMM_WORLD);

      }

   }

   else {

      MPI_Send(&rank, 1, MPI_INT, 0, 0, MPI_COMM_WORLD);

      printf("Process %i sent %i to %i\n", rank, rank, 0);

      MPI_Recv(&msg, 1, MPI_INT, MPI_ANY_SOURCE, MPI_ANY_TAG,
MPI_COMM_WORLD, MPI_STATUS_IGNORE);

      printf("Process %i received %i\n", rank, msg);

   }

   printf("Process %i exiting\n", rank);

   MPI_Finalize();

   return 0;

}

C:\GDX>mpirun -v -display-map -hostfile mpihosts -np 5 c:\gdx\distmsgb.exe

 ======================== JOB MAP ========================

 Data for node: Yap Num procs: 2

        Process OMPI jobid: [43126,1] Process rank: 0

        Process OMPI jobid: [43126,1] Process rank: 1

 Data for node: chuuk Num procs: 2

        Process OMPI jobid: [43126,1] Process rank: 2

        Process OMPI jobid: [43126,1] Process rank: 3

 Data for node: kosrae Num procs: 1

        Process OMPI jobid: [43126,1] Process rank: 4

 =============================================================

connecting to chuuk

username:administrator

password:********

Save Credential?(Y/N) n

connecting to kosrae

username:administrator

password:********

Save Credential?(Y/N) n

Process 0 of 5 initialized

Process 1 of 5 initialized

Process 1 sent 1 to 0

Process 0 received 4

Process 0 received 1

Process 0 received 2

Process 0 received 3

Process 0 sending 1 to 1

Process 0 sending 2 to 2

Process 0 sending 3 to 3

Process 0 sending 4 to 4

Process 0 exiting

Process 1 received 0

Process 1 exiting