Jeff Squyres wrote:
> (for the web archives)
>
> Brock and I talked about this .f90 code a bit off list -- he's going
> to investigate with the test author a bit more because both of us are
> a bit confused by the F90 array syntax used.
Attached is a simple send/recv code written (procedural) C++ that
illustrates a similar problem. It dies at a random number of iterations
with openmpi-1.3.2 or .3. (I have submitted this before.) On some
machines
this goes away with the "-mca btl_sm_num_fifos 8" or
"-mca btl ^sm", so I think this is
https://svn.open-mpi.org/trac/ompi/ticket/2043.
Since it has barriers after each send/recv pair, I do not understand how
any buffers could fill up.
Various stats:
iter.cary$ uname -a
Linux iter.txcorp.com 2.6.29.4-167.fc11.x86_64 #1 SMP Wed May 27
17:27:08 EDT 2009 x86_64 x86_64 x86_64 GNU/Linux
iter.cary$ g++ --version
g++ (GCC) 4.4.0 20090506 (Red Hat 4.4.0-4)
Copyright (C) 2009 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
iter.cary$ mpicxx -show
g++ -I/usr/local/openmpi-1.3.2-nodlopen/include -pthread
-L/usr/local/torque-2.4.0b1/lib -Wl,--rpath
-Wl,/usr/local/torque-2.4.0b1/lib
-Wl,-rpath,/usr/local/openmpi-1.3.2-nodlopen/lib
-L/usr/local/openmpi-1.3.2-nodlopen/lib -lmpi_cxx -lmpi -lopen-rte
-lopen-pal -ltorque -ldl -lnsl -lutil -lm
Seen failures on 64 bit hardware only.
John Cary
>
>
>
> On Dec 1, 2009, at 10:46 AM, Brock Palen wrote:
>
>> The attached code, is an example where openmpi/1.3.2 will lock up, if
>> ran on 48 cores, of IB (4 cores per node),
>> The code loops over recv from all processors on rank 0 and sends from
>> all other ranks, as far as I know this should work, and I can't see
>> why not.
>> Note yes I know we can do the same thing with a gather, this is a
>> simple case to demonstrate the issue.
>> Note that if I increase the openib eager limit, the program runs,
>> which normally means improper MPI, but I can't on my own figure out
>> the problem with this code.
>>
>> Any input on why code like this locks up, unless we up the eager
>> buffer would be helpful, as we should be be having to up the buffer
>> size, just to make code run, makes me feel hacky and dirty.
>>
>>
>> <sendbuf.f90><ATT9198877.txt><ATT9198879.txt>
>
>
/**
* A simple test program to demonstrate a problem in OpenMPI 1.3
*
* Make with:
* mpicxx -o ompi1.3.3-bug ompi1.3.3-bug.cxx
*
* Run with:
* mpirun -n 3 ompi1.3.3-bug
*/
// mpi includes
#include <mpi.h>
// std includes
#include <iostream>
#include <vector>
// useful hashdefine
#define ARRAY_SIZE 250
/**
* Main driver
*/
int main(int argc, char** argv) {
// Initialize MPI
MPI_Init(&argc, &argv);
int rk, sz;
MPI_Comm_rank(MPI_COMM_WORLD, &rk);
MPI_Comm_size(MPI_COMM_WORLD, &sz);
// Create some data to pass around
std::vector<double> d(ARRAY_SIZE);
// Initialize to some values if we aren't rank 0
if ( rk )
for ( unsigned i = 0; i < ARRAY_SIZE; ++i )
d[i] = 2*i + 1;
// Loop until this breaks
unsigned t = 0;
while ( 1 ) {
MPI_Status s;
if ( rk )
MPI_Send( &d[0], d.size(), MPI_DOUBLE, 0, 3, MPI_COMM_WORLD );
else
for ( int i = 1; i < sz; ++i )
MPI_Recv( &d[0], d.size(), MPI_DOUBLE, i, 3, MPI_COMM_WORLD, &s );
MPI_Barrier(MPI_COMM_WORLD);
std::cout << "Transmission " << ++t << " completed." << std::endl;
}
// Finalize MPI
MPI_Finalize();
}
|
![[Flag]](../../../../../community/mirrors/flag-us.gif){kind=small}