Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: Re: [OMPI users] Program deadlocks, on simple send/recv loop
From: John R. Cary (cary_at_[hidden])
Date: 2009-12-01 20:09:31


Jeff Squyres wrote:
> (for the web archives)
>
> Brock and I talked about this .f90 code a bit off list -- he's going
> to investigate with the test author a bit more because both of us are
> a bit confused by the F90 array syntax used.
Attached is a simple send/recv code written (procedural) C++ that
illustrates a similar problem. It dies at a random number of iterations
with openmpi-1.3.2 or .3. (I have submitted this before.) On some
machines
this goes away with the "-mca btl_sm_num_fifos 8" or
"-mca btl ^sm", so I think this is
https://svn.open-mpi.org/trac/ompi/ticket/2043.

Since it has barriers after each send/recv pair, I do not understand how
any buffers could fill up.

Various stats:

iter.cary$ uname -a
Linux iter.txcorp.com 2.6.29.4-167.fc11.x86_64 #1 SMP Wed May 27
17:27:08 EDT 2009 x86_64 x86_64 x86_64 GNU/Linux
iter.cary$ g++ --version
g++ (GCC) 4.4.0 20090506 (Red Hat 4.4.0-4)
Copyright (C) 2009 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

iter.cary$ mpicxx -show
g++ -I/usr/local/openmpi-1.3.2-nodlopen/include -pthread
-L/usr/local/torque-2.4.0b1/lib -Wl,--rpath
-Wl,/usr/local/torque-2.4.0b1/lib
-Wl,-rpath,/usr/local/openmpi-1.3.2-nodlopen/lib
-L/usr/local/openmpi-1.3.2-nodlopen/lib -lmpi_cxx -lmpi -lopen-rte
-lopen-pal -ltorque -ldl -lnsl -lutil -lm

Seen failures on 64 bit hardware only.

John Cary
>
>
>
> On Dec 1, 2009, at 10:46 AM, Brock Palen wrote:
>
>> The attached code, is an example where openmpi/1.3.2 will lock up, if
>> ran on 48 cores, of IB (4 cores per node),
>> The code loops over recv from all processors on rank 0 and sends from
>> all other ranks, as far as I know this should work, and I can't see
>> why not.
>> Note yes I know we can do the same thing with a gather, this is a
>> simple case to demonstrate the issue.
>> Note that if I increase the openib eager limit, the program runs,
>> which normally means improper MPI, but I can't on my own figure out
>> the problem with this code.
>>
>> Any input on why code like this locks up, unless we up the eager
>> buffer would be helpful, as we should be be having to up the buffer
>> size, just to make code run, makes me feel hacky and dirty.
>>
>>
>> <sendbuf.f90><ATT9198877.txt><ATT9198879.txt>
>
>


/**
 * A simple test program to demonstrate a problem in OpenMPI 1.3
 *
 * Make with:
 * mpicxx -o ompi1.3.3-bug ompi1.3.3-bug.cxx
 *
 * Run with:
 * mpirun -n 3 ompi1.3.3-bug
 */

// mpi includes
#include <mpi.h>

// std includes
#include <iostream>
#include <vector>

// useful hashdefine
#define ARRAY_SIZE 250

/**
 * Main driver
 */
int main(int argc, char** argv) {
// Initialize MPI
  MPI_Init(&argc, &argv);

  int rk, sz;
  MPI_Comm_rank(MPI_COMM_WORLD, &rk);
  MPI_Comm_size(MPI_COMM_WORLD, &sz);

// Create some data to pass around
  std::vector<double> d(ARRAY_SIZE);

// Initialize to some values if we aren't rank 0
  if ( rk )
    for ( unsigned i = 0; i < ARRAY_SIZE; ++i )
      d[i] = 2*i + 1;

// Loop until this breaks
  unsigned t = 0;
  while ( 1 ) {
    MPI_Status s;
    if ( rk )
      MPI_Send( &d[0], d.size(), MPI_DOUBLE, 0, 3, MPI_COMM_WORLD );
    else
      for ( int i = 1; i < sz; ++i )
        MPI_Recv( &d[0], d.size(), MPI_DOUBLE, i, 3, MPI_COMM_WORLD, &s );
    MPI_Barrier(MPI_COMM_WORLD);
    std::cout << "Transmission " << ++t << " completed." << std::endl;
  }

// Finalize MPI
  MPI_Finalize();
}