Hello,

I have run into an issue that appears to be related to sending messages to multiple processes on a single remote host prior to the remote processes sending messages to the origin. I have cooked the issue down to the following:

Test Environment of 3 Identical Hosts:

·         * Intel i7-2600K, 12GB ram, Intel GB Ethernet, DLink Switch

 ·       * Windows 2008R2 x64 with all current updates

·         * OMPI  (all three hosts report the same ompi_info and were installed with the same binary)
http://www.open-mpi.org/software/ompi/v1.5/downloads/OpenMPI_v1.5.4-1_win64.exe

C:\GDX>ompi_info -v ompi full --parsable

package:Open MPI hpcfan@VISCLUSTER26 Distribution

ompi:version:full:1.5.4

ompi:version:svn:r25060

ompi:version:release_date:Aug 18, 2011

orte:version:full:1.5.4

orte:version:svn:r25060

orte:version:release_date:Aug 18, 2011

opal:version:full:1.5.4

opal:version:svn:r25060

opal:version:release_date:Aug 18, 2011

ident:1.5.4

Test Program:

#include <stdio.h>

#define OMPI_IMPORTS

#include "C:\Program Files (x86)\OpenMPI_v1.5.4-x64\include\mpi.h"

int main(int argc, char *argv[])

{

   int rank, size, i, msg;

   MPI_Init(&argc, &argv);

   MPI_Comm_rank(MPI_COMM_WORLD, &rank);

   MPI_Comm_size(MPI_COMM_WORLD, &size);

   printf("Process %i of %i initialized\n", rank, size);

   if (0 == rank) {

      for (i = 1; i < size; i++) {

         printf("Process %i sending %i to %i\n", rank, i, i);

         MPI_Send(&rank, 1, MPI_INT, i, 0, MPI_COMM_WORLD);

      }

      for (i = 1; i < size; i++) {

         MPI_Recv(&msg, 1, MPI_INT, MPI_ANY_SOURCE, MPI_ANY_TAG, MPI_COMM_WORLD, MPI_STATUS_IGNORE);

         printf("Process %i received %i\n", rank, msg);

      }

   }

   else {

      MPI_Recv(&msg, 1, MPI_INT, MPI_ANY_SOURCE, MPI_ANY_TAG, MPI_COMM_WORLD, MPI_STATUS_IGNORE);

      printf("Process %i received %i\n", rank, msg);

      MPI_Send(&rank, 1, MPI_INT, 0, 0, MPI_COMM_WORLD);

      printf("Process %i sent %i to %i\n", rank, rank, 0);

   }

   printf("Process %i exiting\n", rank);

   MPI_Finalize();

   return 0;

}

Test Cases:

·            X procs on the originating node: Working

·            X procs on the originating node and one proc on one or more remote nodes: Working

·            X procs on the originating node and more than one proc on any remote node:  Fails

A test with two procs on the origin and one on each of two remote nodes runs, however the same test with the two remote procs on the same machine fails on the second remote send. Here are some test runs (the ^C indicates a hang).

C:\GDX>mpirun -v -display-map -hostfile mpihosts -np 2 c:\gdx\distmsg.exe

 ========================   JOB MAP   ========================

 Data for node: Yap     Num procs: 2

        Process OMPI jobid: [42094,1] Process rank: 0

        Process OMPI jobid: [42094,1] Process rank: 1

 =============================================================

Process 0 of 2 initialized

Process 1 of 2 initialized

Process 0 sending 1 to 1

Process 1 received 0

Process 1 sent 1 to 0

Process 1 exiting

Process 0 received 1

Process 0 exiting

C:\GDX>mpirun -v -display-map -hostfile mpihosts -np 3 c:\gdx\distmsg.exe

 ========================   JOB MAP   ========================

 Data for node: Yap     Num procs: 2

        Process OMPI jobid: [42014,1] Process rank: 0

        Process OMPI jobid: [42014,1] Process rank: 1

 Data for node: chuuk   Num procs: 1

        Process OMPI jobid: [42014,1] Process rank: 2

 =============================================================

connecting to chuuk

username:administrator

password:********

Save Credential?(Y/N) n

Process 0 of 3 initialized

Process 1 of 3 initialized

Process 0 sending 1 to 1

Process 0 sending 2 to 2

Process 1 received 0

Process 1 sent 1 to 0

Process 1 exiting

Process 0 received 1

Process 0 received 2

Process 0 exiting

C:\GDX>mpirun -v -display-map -hostfile mpihosts -np 4 c:\gdx\distmsg.exe

 ========================   JOB MAP   ========================

 Data for node: Yap     Num procs: 2

        Process OMPI jobid: [43894,1] Process rank: 0

        Process OMPI jobid: [43894,1] Process rank: 1

 Data for node: chuuk   Num procs: 2

        Process OMPI jobid: [43894,1] Process rank: 2

        Process OMPI jobid: [43894,1] Process rank: 3

 =============================================================

connecting to chuuk

username:administrator

password:********

Save Credential?(Y/N) n

Process 0 of 4 initialized

Process 1 of 4 initialized

Process 0 sending 1 to 1

Process 0 sending 2 to 2

Process 1 received 0

Process 1 sent 1 to 0

Process 1 exiting

Process 0 sending 3 to 3

^C

C:\GDX>mpirun -v -display-map -hostfile mpihosts -np 4 c:\gdx\distmsg.exe

 ========================   JOB MAP   ========================

 Data for node: Yap     Num procs: 2

        Process OMPI jobid: [43310,1] Process rank: 0

        Process OMPI jobid: [43310,1] Process rank: 1

 Data for node: chuuk   Num procs: 1

        Process OMPI jobid: [43310,1] Process rank: 2

 Data for node: kosrae  Num procs: 1

        Process OMPI jobid: [43310,1] Process rank: 3

 =============================================================

connecting to chuuk

username:administrator

password:********

Save Credential?(Y/N) n

connecting to kosrae

username:administrator

password:********

Save Credential?(Y/N) n

Process 0 of 4 initialized

Process 1 of 4 initialized

Process 0 sending 1 to 1

Process 0 sending 2 to 2

Process 1 received 0

Process 1 sent 1 to 0

Process 1 exiting

Process 0 sending 3 to 3

Process 0 received 1

Process 0 received 2

Process 0 received 3

Process 0 exiting

C:\GDX>mpirun -v -display-map -hostfile mpihosts -np 5 c:\gdx\distmsg.exe

 ========================   JOB MAP   ========================

 Data for node: Yap     Num procs: 2

        Process OMPI jobid: [43590,1] Process rank: 0

        Process OMPI jobid: [43590,1] Process rank: 1

 Data for node: chuuk   Num procs: 2

        Process OMPI jobid: [43590,1] Process rank: 2

        Process OMPI jobid: [43590,1] Process rank: 3

 Data for node: kosrae  Num procs: 1

        Process OMPI jobid: [43590,1] Process rank: 4

 =============================================================

connecting to chuuk

username:administrator

password:********

Save Credential?(Y/N) n

connecting to kosrae

username:administrator

password:********

Save Credential?(Y/N) n

Process 0 of 5 initialized

Process 1 of 5 initialized

Process 0 sending 1 to 1

Process 0 sending 2 to 2

Process 1 received 0

Process 1 sent 1 to 0

Process 1 exiting

Process 0 sending 3 to 3

^C

The remote process which is the target of the hung send seems to generate significant ongoing CPU activity and "Other" I/O.

Workaround

Curiously swapping the send/receive order solves the problem.

#include <stdio.h>

#define OMPI_IMPORTS

#include "C:\Program Files (x86)\OpenMPI_v1.5.4-x64\include\mpi.h"

int main(int argc, char *argv[])

{

   int rank, size, i, msg;

   MPI_Init(&argc, &argv);

   MPI_Comm_rank(MPI_COMM_WORLD, &rank);

   MPI_Comm_size(MPI_COMM_WORLD, &size);

   printf("Process %i of %i initialized\n", rank, size);

   if (0 == rank) {

      for (i = 1; i < size; i++) {

         MPI_Recv(&msg, 1, MPI_INT, MPI_ANY_SOURCE, MPI_ANY_TAG, MPI_COMM_WORLD, MPI_STATUS_IGNORE);

         printf("Process %i received %i\n", rank, msg);

      }

      for (i = 1; i < size; i++) {

         printf("Process %i sending %i to %i\n", rank, i, i);

         MPI_Send(&rank, 1, MPI_INT, i, 0, MPI_COMM_WORLD);

      }

   }

   else {

      MPI_Send(&rank, 1, MPI_INT, 0, 0, MPI_COMM_WORLD);

      printf("Process %i sent %i to %i\n", rank, rank, 0);

      MPI_Recv(&msg, 1, MPI_INT, MPI_ANY_SOURCE, MPI_ANY_TAG, MPI_COMM_WORLD, MPI_STATUS_IGNORE);

      printf("Process %i received %i\n", rank, msg);

   }

   printf("Process %i exiting\n", rank);

   MPI_Finalize();

   return 0;

}

C:\GDX>mpirun -v -display-map -hostfile mpihosts -np 5 c:\gdx\distmsgb.exe

 ========================   JOB MAP   ========================

 Data for node: Yap     Num procs: 2

        Process OMPI jobid: [43126,1] Process rank: 0

        Process OMPI jobid: [43126,1] Process rank: 1

 Data for node: chuuk   Num procs: 2

        Process OMPI jobid: [43126,1] Process rank: 2

        Process OMPI jobid: [43126,1] Process rank: 3

 Data for node: kosrae  Num procs: 1

        Process OMPI jobid: [43126,1] Process rank: 4

 =============================================================

connecting to chuuk

username:administrator

password:********

Save Credential?(Y/N) n

connecting to kosrae

username:administrator

password:********

Save Credential?(Y/N) n

Process 0 of 5 initialized

Process 1 of 5 initialized

Process 1 sent 1 to 0

Process 0 received 4

Process 0 received 1

Process 0 received 2

Process 0 received 3

Process 0 sending 1 to 1

Process 0 sending 2 to 2

Process 0 sending 3 to 3

Process 0 sending 4 to 4

Process 0 exiting

Process 1 received 0

Process 1 exiting