Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: [OMPI users] Openmpi failure on dual quad linux
From: John R. Cary (cary_at_[hidden])
Date: 2009-10-10 12:09:47


We have been getting hangs and failures of openmpi-1.3.X
on an 8-core FC11 box. Details:

Machine:

Linux octet.carys.home 2.6.30.5-43.fc11.x86_64 #1 SMP Thu Aug 27
21:39:52 EDT 2009 x86_64 x86_64 x86_64 GNU/Linux

with 2 quad-core cpus.

Compiler:

g++ (GCC) 4.4.1 20090725 (Red Hat 4.4.1-2)

The code is later in this email.

OpenMPI configured with

../configure --prefix=/usr/local/openmpi-1.3.3
--with-wrapper-ldflags=-Wl,-rpath,/usr/local/openmpi-1.3.3/lib

used to compile and execute:

/usr/local/openmpi-1.3.3/bin/mpicxx ompi1.3.3-bug.cxx -o ompi1.3.3-bug
/usr/local/openmpi-1.3.3/bin/mpirun -np 8 ompi1.3.3-bug

hangs on various steps: 10, 39, 125. Top shows the code
using all resources:

  PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+
COMMAND
 5716 cary 20 0 98.6m 4396 2940 R 100.3 0.0 0:33.58
ompi1.3.3-bug
 5707 cary 20 0 98.7m 4592 3060 R 99.9 0.0 0:33.57
ompi1.3.3-bug
 5710 cary 20 0 98.6m 4388 2932 R 99.9 0.0 0:33.61
ompi1.3.3-bug
 5713 cary 20 0 98.6m 4392 2936 R 99.9 0.0 0:33.61
ompi1.3.3-bug
 5719 cary 20 0 98.6m 4396 2940 R 99.9 0.0 0:33.60
ompi1.3.3-bug
 5722 cary 20 0 98.6m 4384 2928 R 99.9 0.0 0:33.61
ompi1.3.3-bug
 5728 cary 20 0 98.6m 4408 2952 R 99.9 0.0 0:33.53
ompi1.3.3-bug
 5725 cary 20 0 98.6m 4404 2948 R 99.6 0.0 0:33.57 ompi1.3.3-bug

OpenMPI configured with

../configure --prefix=/usr/local/openmpi-1.2.9
--with-wrapper-ldflags=-Wl,-rpath,/usr/local/openmpi-1.2.9/lib

used to compile and execute:

/usr/local/openmpi-1.2.9/bin/mpicxx ompi1.3.3-bug.cxx -o ompi1.3.3-bug
/usr/local/openmpi-1.2.9/bin/mpirun -np 8 ompi1.3.3-bug

works just fine.

So then I started looking for unitialized variables. I rebuilt
OpenMPI, configured with

../configure --prefix=/usr/local/openmpi-1.3.3-valg
--with-valgrind=/contrib/valgrind
--with-wrapper-ldflags=-Wl,-rpath,/usr/local/openmpi-1.3.3-valg/lib

and used to to compile and execute:

/usr/local/openmpi-1.3.3-valg/bin/mpicxx ompi1.3.3-bug.cxx -o ompi1.3.3-bug
/usr/local/openmpi-1.3.3-valg/bin/mpirun -np 8 valgrind ompi1.3.3-bug

and one can get valgrind to segfault, as shown at the very end of this
email. Other times it hangs on step 1 or another step.

John Cary

THE CODE

$ cat ompi1.3.3-bug.cxx

/**
 * A simple test program to demonstrate a problem in OpenMPI 1.3
 */

// mpi includes
#include <mpi.h>

// std includes
#include <iostream>
#include <vector>

// useful hashdefine
#define ARRAY_SIZE 250

/**
 * Main driver
 */
int main(int argc, char** argv) {
// Initialize MPI
  MPI_Init(&argc, &argv);

  int rk, sz;
  MPI_Comm_rank(MPI_COMM_WORLD, &rk);
  MPI_Comm_size(MPI_COMM_WORLD, &sz);

// Create some data to pass around
  std::vector<double> d(ARRAY_SIZE);

// Initialize to some values if we aren't rank 0
  if ( rk )
    for ( unsigned i = 0; i < ARRAY_SIZE; ++i )
      d[i] = 2*i + 1;

// Loop until this breaks
  unsigned t = 0;
  while ( 1 ) {
    MPI_Status s;
    if ( rk )
      MPI_Send( &d[0], d.size(), MPI_DOUBLE, 0, 3, MPI_COMM_WORLD );
    else
      for ( int i = 1; i < sz; ++i )
        MPI_Recv( &d[0], d.size(), MPI_DOUBLE, i, 3, MPI_COMM_WORLD, &s );
    MPI_Barrier(MPI_COMM_WORLD);
    std::cout << "Transmission " << ++t << " completed." << std::endl;
  }

// Finalize MPI
  MPI_Finalize();
}

THE VALGRIND RUN

octet.cary$ /usr/local/openmpi-1.3.3-valg/bin/mpirun -np 8 valgrind
ompi1.3.3-bug
==902== Memcheck, a memory error detector
==902== Copyright (C) 2002-2009, and GNU GPL'd, by Julian Seward et al.
==902== Using Valgrind-3.5.0 and LibVEX; rerun with -h for copyright info
==902== Command: ompi1.3.3-bug
==902==
==905== Memcheck, a memory error detector
==905== Copyright (C) 2002-2009, and GNU GPL'd, by Julian Seward et al.
==905== Using Valgrind-3.5.0 and LibVEX; rerun with -h for copyright info
==905== Command: ompi1.3.3-bug
==905==
==908== Memcheck, a memory error detector
==908== Copyright (C) 2002-2009, and GNU GPL'd, by Julian Seward et al.
==908== Using Valgrind-3.5.0 and LibVEX; rerun with -h for copyright info
==908== Command: ompi1.3.3-bug
==908==
==911== Memcheck, a memory error detector
==911== Copyright (C) 2002-2009, and GNU GPL'd, by Julian Seward et al.
==911== Using Valgrind-3.5.0 and LibVEX; rerun with -h for copyright info
==911== Command: ompi1.3.3-bug
==911==
==914== Memcheck, a memory error detector
==914== Copyright (C) 2002-2009, and GNU GPL'd, by Julian Seward et al.
==914== Using Valgrind-3.5.0 and LibVEX; rerun with -h for copyright info
==914== Command: ompi1.3.3-bug
==914==
==917== Memcheck, a memory error detector
==917== Copyright (C) 2002-2009, and GNU GPL'd, by Julian Seward et al.
==917== Using Valgrind-3.5.0 and LibVEX; rerun with -h for copyright info
==917== Command: ompi1.3.3-bug
==917==
==920== Memcheck, a memory error detector
==920== Copyright (C) 2002-2009, and GNU GPL'd, by Julian Seward et al.
==920== Using Valgrind-3.5.0 and LibVEX; rerun with -h for copyright info
==920== Command: ompi1.3.3-bug
==920==
==923== Memcheck, a memory error detector
==923== Copyright (C) 2002-2009, and GNU GPL'd, by Julian Seward et al.
==923== Using Valgrind-3.5.0 and LibVEX; rerun with -h for copyright info
==923== Command: ompi1.3.3-bug
==923==
==917== Warning: client syscall munmap tried to modify addresses
0xffffffffffffffff-0xffe
==902== Warning: client syscall munmap tried to modify addresses
0xffffffffffffffff-0xffe
==905== Warning: client syscall munmap tried to modify addresses
0xffffffffffffffff-0xffe
==914== Warning: client syscall munmap tried to modify addresses
0xffffffffffffffff-0xffe
==911== Warning: client syscall munmap tried to modify addresses
0xffffffffffffffff-0xffe
==920== Warning: client syscall munmap tried to modify addresses
0xffffffffffffffff-0xffe
==923== Warning: client syscall munmap tried to modify addresses
0xffffffffffffffff-0xffe
==908== Warning: client syscall munmap tried to modify addresses
0xffffffffffffffff-0xffe
==902== Conditional jump or move depends on uninitialised value(s)
==902== at 0x7BDFD64: mca_mpool_sm_alloc (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)
==902== by 0x8C1F94D: mca_btl_sm_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)
==902== by 0x8816759: mca_bml_r2_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)
==902== by 0x8400274: mca_pml_ob1_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)
==902== by 0x4E5B048: ompi_mpi_init (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==902== by 0x4E79F65: PMPI_Init (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==902== by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
==902==
==902== Conditional jump or move depends on uninitialised value(s)
==902== at 0x7BDFD64: mca_mpool_sm_alloc (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)
==902== by 0x4E3EBDF: ompi_free_list_grow (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==902== by 0x8C1FA22: mca_btl_sm_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)
==902== by 0x8816759: mca_bml_r2_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)
==902== by 0x8400274: mca_pml_ob1_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)
==902== by 0x4E5B048: ompi_mpi_init (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==902== by 0x4E79F65: PMPI_Init (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==902== by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
==902==
==911== Conditional jump or move depends on uninitialised value(s)
==911== at 0x7BDFD64: mca_mpool_sm_alloc (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)
==911== by 0x8C1F94D: mca_btl_sm_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)
==911== by 0x8816759: mca_bml_r2_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)
==911== by 0x8400274: mca_pml_ob1_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)
==911== by 0x4E5B048: ompi_mpi_init (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==911== by 0x4E79F65: PMPI_Init (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==911== by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
==911==
==902== Conditional jump or move depends on uninitialised value(s)
==902== at 0x7BDFD64: mca_mpool_sm_alloc (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)
==902== by 0x4E3EBDF: ompi_free_list_grow (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==902== by 0x8C1FA93: mca_btl_sm_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)
==902== by 0x8816759: mca_bml_r2_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)
==902== by 0x8400274: mca_pml_ob1_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)
==902== by 0x4E5B048: ompi_mpi_init (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==902== by 0x4E79F65: PMPI_Init (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==914== Conditional jump or move depends on uninitialised value(s)
==902== by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
==902==
==914== at 0x7BDFD64: mca_mpool_sm_alloc (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)
==914== by 0x8C1F94D: mca_btl_sm_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)
==914== by 0x8816759: mca_bml_r2_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)
==914== by 0x8400274: mca_pml_ob1_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)
==914== by 0x4E5B048: ompi_mpi_init (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==914== by 0x4E79F65: PMPI_Init (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==914== by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
==914==
==917== Conditional jump or move depends on uninitialised value(s)
==917== at 0x7BDFD64: mca_mpool_sm_alloc (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)
==917== by 0x8C1F94D: mca_btl_sm_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)
==917== by 0x8816759: mca_bml_r2_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)
==917== by 0x8400274: mca_pml_ob1_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)
==917== by 0x4E5B048: ompi_mpi_init (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==917== by 0x4E79F65: PMPI_Init (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==917== by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
==917==
==911== Conditional jump or move depends on uninitialised value(s)
==911== at 0x7BDFD64: mca_mpool_sm_alloc (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)
==911== by 0x4E3EBDF: ompi_free_list_grow (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==911== by 0x8C1FA22: mca_btl_sm_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)
==911== by 0x8816759: mca_bml_r2_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)
==911== by 0x8400274: mca_pml_ob1_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)
==911== by 0x4E5B048: ompi_mpi_init (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==911== by 0x4E79F65: PMPI_Init (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==911== by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
==911==
==914== Conditional jump or move depends on uninitialised value(s)
==914== at 0x7BDFD64: mca_mpool_sm_alloc (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)
==914== by 0x4E3EBDF: ompi_free_list_grow (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==914== by 0x8C1FA22: mca_btl_sm_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)
==914== by 0x8816759: mca_bml_r2_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)
==914== by 0x8400274: mca_pml_ob1_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)
==914== by 0x4E5B048: ompi_mpi_init (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==914== by 0x4E79F65: PMPI_Init (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==914== by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
==914==
==908== Conditional jump or move depends on uninitialised value(s)
==908== at 0x7BDFD64: mca_mpool_sm_alloc (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)
==908== by 0x8C1F94D: mca_btl_sm_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)
==908== by 0x8816759: mca_bml_r2_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)
==908== by 0x8400274: mca_pml_ob1_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)
==908== by 0x4E5B048: ompi_mpi_init (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==908== by 0x4E79F65: PMPI_Init (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==908== by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
==908==
==902== Conditional jump or move depends on uninitialised value(s)
==902== at 0x7BDFD64: mca_mpool_sm_alloc (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)
==902== by 0x8C1F0FC: mca_btl_sm_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)
==902== by 0x8816759: mca_bml_r2_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)
==902== by 0x8400274: mca_pml_ob1_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)
==902== by 0x4E5B048: ompi_mpi_init (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==902== by 0x4E79F65: PMPI_Init (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==902== by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
==902==
==911== Conditional jump or move depends on uninitialised value(s)
==911== at 0x7BDFD64: mca_mpool_sm_alloc (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)
==911== by 0x4E3EBDF: ompi_free_list_grow (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==923== Conditional jump or move depends on uninitialised value(s)
==911== by 0x8C1FA93: mca_btl_sm_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)
==911== by 0x8816759: mca_bml_r2_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)
==911== by 0x8400274: mca_pml_ob1_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)
==911== by 0x4E5B048: ompi_mpi_init (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==911== by 0x4E79F65: PMPI_Init (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==911== by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
==911==
==923== at 0x7BDFD64: mca_mpool_sm_alloc (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)
==923== by 0x8C1F94D: mca_btl_sm_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)
==920== Conditional jump or move depends on uninitialised value(s)
==923== by 0x8816759: mca_bml_r2_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)
==923== by 0x8400274: mca_pml_ob1_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)
==923== by 0x4E5B048: ompi_mpi_init (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==923== by 0x4E79F65: PMPI_Init (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==923== by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
==923==
==920== at 0x7BDFD64: mca_mpool_sm_alloc (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)
==920== by 0x8C1F94D: mca_btl_sm_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)
==920== by 0x8816759: mca_bml_r2_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)
==920== by 0x8400274: mca_pml_ob1_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)
==920== by 0x4E5B048: ompi_mpi_init (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==920== by 0x4E79F65: PMPI_Init (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==920== by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
==920==
==914== Conditional jump or move depends on uninitialised value(s)
==914== at 0x7BDFD64: mca_mpool_sm_alloc (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)
==914== by 0x4E3EBDF: ompi_free_list_grow (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==914== by 0x8C1FA93: mca_btl_sm_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)
==914== by 0x8816759: mca_bml_r2_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)
==914== by 0x8400274: mca_pml_ob1_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)
==914== by 0x4E5B048: ompi_mpi_init (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==914== by 0x4E79F65: PMPI_Init (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==914== by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
==914==
==917== Conditional jump or move depends on uninitialised value(s)
==917== at 0x7BDFD64: mca_mpool_sm_alloc (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)
==917== by 0x4E3EBDF: ompi_free_list_grow (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==917== by 0x8C1FA22: mca_btl_sm_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)
==917== by 0x8816759: mca_bml_r2_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)
==917== by 0x8400274: mca_pml_ob1_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)
==917== by 0x4E5B048: ompi_mpi_init (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==917== by 0x4E79F65: PMPI_Init (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==917== by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
==917==
==917== Conditional jump or move depends on uninitialised value(s)
==908== Conditional jump or move depends on uninitialised value(s)
==908== at 0x7BDFD64: mca_mpool_sm_alloc (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)
==908== by 0x4E3EBDF: ompi_free_list_grow (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==908== by 0x8C1FA22: mca_btl_sm_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)
==908== by 0x8816759: mca_bml_r2_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)
==908== by 0x8400274: mca_pml_ob1_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)
==908== by 0x4E5B048: ompi_mpi_init (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==908== by 0x4E79F65: PMPI_Init (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==908== by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
==908==
==917== at 0x7BDFD64: mca_mpool_sm_alloc (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)
==917== by 0x4E3EBDF: ompi_free_list_grow (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==917== by 0x8C1FA93: mca_btl_sm_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)
==917== by 0x8816759: mca_bml_r2_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)
==917== by 0x8400274: mca_pml_ob1_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)
==917== by 0x4E5B048: ompi_mpi_init (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==911== Conditional jump or move depends on uninitialised value(s)
==911== at 0x7BDFD64: mca_mpool_sm_alloc (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)
==911== by 0x8C1F0FC: mca_btl_sm_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)
==911== by 0x8816759: mca_bml_r2_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)
==911== by 0x8400274: mca_pml_ob1_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)
==911== by 0x4E5B048: ompi_mpi_init (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==911== by 0x4E79F65: PMPI_Init (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==911== by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
==911==
==917== by 0x4E79F65: PMPI_Init (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==917== by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
==914== Conditional jump or move depends on uninitialised value(s)
==914== at 0x7BDFD64: mca_mpool_sm_alloc (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)
==914== by 0x8C1F0FC: mca_btl_sm_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)
==914== by 0x8816759: mca_bml_r2_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)
==914== by 0x8400274: mca_pml_ob1_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)
==914== by 0x4E5B048: ompi_mpi_init (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==914== by 0x4E79F65: PMPI_Init (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==914== by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
==914==
==917==
==923== Conditional jump or move depends on uninitialised value(s)
==923== at 0x7BDFD64: mca_mpool_sm_alloc (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)
==923== by 0x4E3EBDF: ompi_free_list_grow (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==923== by 0x8C1FA22: mca_btl_sm_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)
==923== by 0x8816759: mca_bml_r2_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)
==923== by 0x8400274: mca_pml_ob1_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)
==923== by 0x4E5B048: ompi_mpi_init (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==923== by 0x4E79F65: PMPI_Init (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==923== by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
==923==
==920== Conditional jump or move depends on uninitialised value(s)
==920== at 0x7BDFD64: mca_mpool_sm_alloc (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)
==920== by 0x4E3EBDF: ompi_free_list_grow (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==920== by 0x8C1FA22: mca_btl_sm_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)
==920== by 0x8816759: mca_bml_r2_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)
==920== by 0x8400274: mca_pml_ob1_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)
==920== by 0x4E5B048: ompi_mpi_init (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==920== by 0x4E79F65: PMPI_Init (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==920== by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
==920==
==908== Conditional jump or move depends on uninitialised value(s)
==908== at 0x7BDFD64: mca_mpool_sm_alloc (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)
==908== by 0x4E3EBDF: ompi_free_list_grow (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==908== by 0x8C1FA93: mca_btl_sm_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)
==908== by 0x8816759: mca_bml_r2_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)
==908== by 0x8400274: mca_pml_ob1_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)
==908== by 0x4E5B048: ompi_mpi_init (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==908== by 0x4E79F65: PMPI_Init (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==908== by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
==908==
==923== Conditional jump or move depends on uninitialised value(s)
==923== at 0x7BDFD64: mca_mpool_sm_alloc (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)
==923== by 0x4E3EBDF: ompi_free_list_grow (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==920== Conditional jump or move depends on uninitialised value(s)
==920== at 0x7BDFD64: mca_mpool_sm_alloc (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)
==920== by 0x4E3EBDF: ompi_free_list_grow (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==920== by 0x8C1FA93: mca_btl_sm_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)
==923== by 0x8C1FA93: mca_btl_sm_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)
==923== by 0x8816759: mca_bml_r2_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)
==923== by 0x8400274: mca_pml_ob1_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)
==923== by 0x4E5B048: ompi_mpi_init (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==923== by 0x4E79F65: PMPI_Init (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==923== by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
==923==
==920== by 0x8816759: mca_bml_r2_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)
==920== by 0x8400274: mca_pml_ob1_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)
==920== by 0x4E5B048: ompi_mpi_init (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==920== by 0x4E79F65: PMPI_Init (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==920== by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
==920==
==905== Conditional jump or move depends on uninitialised value(s)
==905== at 0x7BDFD64: mca_mpool_sm_alloc (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)
==905== by 0x8C1F94D: mca_btl_sm_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)
==905== by 0x8816759: mca_bml_r2_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)
==905== by 0x8400274: mca_pml_ob1_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)
==905== by 0x4E5B048: ompi_mpi_init (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==905== by 0x4E79F65: PMPI_Init (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==905== by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
==905==
==917== Conditional jump or move depends on uninitialised value(s)
==917== at 0x7BDFD64: mca_mpool_sm_alloc (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)
==917== by 0x8C1F0FC: mca_btl_sm_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)
==917== by 0x8816759: mca_bml_r2_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)
==917== by 0x8400274: mca_pml_ob1_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)
==917== by 0x4E5B048: ompi_mpi_init (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==917== by 0x4E79F65: PMPI_Init (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==917== by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
==917==
==908== Conditional jump or move depends on uninitialised value(s)
==908== at 0x7BDFD64: mca_mpool_sm_alloc (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)
==908== by 0x8C1F0FC: mca_btl_sm_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)
==908== by 0x8816759: mca_bml_r2_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)
==908== by 0x8400274: mca_pml_ob1_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)
==908== by 0x4E5B048: ompi_mpi_init (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==908== by 0x4E79F65: PMPI_Init (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==908== by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
==908==
==905== Conditional jump or move depends on uninitialised value(s)
==905== at 0x7BDFD64: mca_mpool_sm_alloc (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)
==905== by 0x4E3EBDF: ompi_free_list_grow (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==905== by 0x8C1FA22: mca_btl_sm_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)
==905== by 0x8816759: mca_bml_r2_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)
==905== by 0x8400274: mca_pml_ob1_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)
==905== by 0x4E5B048: ompi_mpi_init (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==905== by 0x4E79F65: PMPI_Init (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==905== by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
==905==
==923== Conditional jump or move depends on uninitialised value(s)
==923== at 0x7BDFD64: mca_mpool_sm_alloc (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)
==923== by 0x8C1F0FC: mca_btl_sm_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)
==923== by 0x8816759: mca_bml_r2_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)
==923== by 0x8400274: mca_pml_ob1_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)
==923== by 0x4E5B048: ompi_mpi_init (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==923== by 0x4E79F65: PMPI_Init (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==923== by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
==923==
==920== Conditional jump or move depends on uninitialised value(s)
==920== at 0x7BDFD64: mca_mpool_sm_alloc (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)
==920== by 0x8C1F0FC: mca_btl_sm_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)
==920== by 0x8816759: mca_bml_r2_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)
==920== by 0x8400274: mca_pml_ob1_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)
==920== by 0x4E5B048: ompi_mpi_init (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==920== by 0x4E79F65: PMPI_Init (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==920== by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
==920==
==905== Conditional jump or move depends on uninitialised value(s)
==905== at 0x7BDFD64: mca_mpool_sm_alloc (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)
==905== by 0x4E3EBDF: ompi_free_list_grow (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==905== by 0x8C1FA93: mca_btl_sm_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)
==905== by 0x8816759: mca_bml_r2_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)
==905== by 0x8400274: mca_pml_ob1_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)
==905== by 0x4E5B048: ompi_mpi_init (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==905== by 0x4E79F65: PMPI_Init (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==905== by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
==905==
==917== Conditional jump or move depends on uninitialised value(s)
==917== at 0x7BDFD64: mca_mpool_sm_alloc (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)
==917== by 0x4E3EBDF: ompi_free_list_grow (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==917== by 0x4E3EC5F: ompi_free_list_resize (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==917== by 0x8C1F630: mca_btl_sm_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)
==917== by 0x8816759: mca_bml_r2_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)
==917== by 0x8400274: mca_pml_ob1_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)
==917== by 0x4E5B048: ompi_mpi_init (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==917== by 0x4E79F65: PMPI_Init (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==917== by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
==917==
==908== Conditional jump or move depends on uninitialised value(s)
==908== at 0x7BDFD64: mca_mpool_sm_alloc (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)
==908== by 0x4E3EBDF: ompi_free_list_grow (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==908== by 0x4E3EC5F: ompi_free_list_resize (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==908== by 0x8C1F630: mca_btl_sm_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)
==908== by 0x8816759: mca_bml_r2_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)
==908== by 0x8400274: mca_pml_ob1_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)
==908== by 0x4E5B048: ompi_mpi_init (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==908== by 0x4E79F65: PMPI_Init (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==908== by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
==908==
==923== Conditional jump or move depends on uninitialised value(s)
==923== at 0x7BDFD64: mca_mpool_sm_alloc (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)
==923== by 0x4E3EBDF: ompi_free_list_grow (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==923== by 0x4E3EC5F: ompi_free_list_resize (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==923== by 0x8C1F630: mca_btl_sm_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)
==923== by 0x8816759: mca_bml_r2_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)
==923== by 0x8400274: mca_pml_ob1_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)
==923== by 0x4E5B048: ompi_mpi_init (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==923== by 0x4E79F65: PMPI_Init (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==923== by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
==923==
==920== Conditional jump or move depends on uninitialised value(s)
==920== at 0x7BDFD64: mca_mpool_sm_alloc (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)
==920== by 0x4E3EBDF: ompi_free_list_grow (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==920== by 0x4E3EC5F: ompi_free_list_resize (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==920== by 0x8C1F630: mca_btl_sm_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)
==920== by 0x8816759: mca_bml_r2_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)
==920== by 0x8400274: mca_pml_ob1_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)
==920== by 0x4E5B048: ompi_mpi_init (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==920== by 0x4E79F65: PMPI_Init (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==920== by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
==920==
==905== Conditional jump or move depends on uninitialised value(s)
==905== at 0x7BDFD64: mca_mpool_sm_alloc (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)
==905== by 0x8C1F0FC: mca_btl_sm_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)
==905== by 0x8816759: mca_bml_r2_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)
==905== by 0x8400274: mca_pml_ob1_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)
==905== by 0x4E5B048: ompi_mpi_init (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==905== by 0x4E79F65: PMPI_Init (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==905== by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
==905==
==905== Conditional jump or move depends on uninitialised value(s)
==905== at 0x7BDFD64: mca_mpool_sm_alloc (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)
==905== by 0x4E3EBDF: ompi_free_list_grow (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==905== by 0x4E3EC5F: ompi_free_list_resize (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==905== by 0x8C1F630: mca_btl_sm_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)
==905== by 0x8816759: mca_bml_r2_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)
==905== by 0x8400274: mca_pml_ob1_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)
==905== by 0x4E5B048: ompi_mpi_init (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==905== by 0x4E79F65: PMPI_Init (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==905== by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
==905==
==902== Conditional jump or move depends on uninitialised value(s)
==902== at 0x7BDFD64: mca_mpool_sm_alloc (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)
==902== by 0x4E3EBDF: ompi_free_list_grow (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==902== by 0x4E3EC5F: ompi_free_list_resize (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==902== by 0x8C1F630: mca_btl_sm_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)
==902== by 0x8816759: mca_bml_r2_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)
==902== by 0x8400274: mca_pml_ob1_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)
==902== by 0x4E5B048: ompi_mpi_init (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==902== by 0x4E79F65: PMPI_Init (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==902== by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
==902==
==911== Conditional jump or move depends on uninitialised value(s)
==911== at 0x7BDFD64: mca_mpool_sm_alloc (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)
==911== by 0x4E3EBDF: ompi_free_list_grow (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==911== by 0x4E3EC5F: ompi_free_list_resize (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==911== by 0x8C1F630: mca_btl_sm_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)
==911== by 0x8816759: mca_bml_r2_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)
==911== by 0x8400274: mca_pml_ob1_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)
==911== by 0x4E5B048: ompi_mpi_init (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==911== by 0x4E79F65: PMPI_Init (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==911== by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
==911==
==914== Conditional jump or move depends on uninitialised value(s)
==914== at 0x7BDFD64: mca_mpool_sm_alloc (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)
==914== by 0x4E3EBDF: ompi_free_list_grow (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==914== by 0x4E3EC5F: ompi_free_list_resize (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==914== by 0x8C1F630: mca_btl_sm_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)
==914== by 0x8816759: mca_bml_r2_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)
==914== by 0x8400274: mca_pml_ob1_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)
==914== by 0x4E5B048: ompi_mpi_init (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==914== by 0x4E79F65: PMPI_Init (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==914== by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
==914==
[octet:00923] *** Process received signal ***
[octet:00923] Signal: Segmentation fault (11)
[octet:00923] Signal code: Address not mapped (1)
[octet:00923] Failing at address: 0x40
[octet:00923] [ 0] /lib64/libpthread.so.0 [0x3825c0eee0]
[octet:00923] [ 1]
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so [0x8c21009]
[octet:00923] [ 2]
/usr/local/openmpi-1.3.3-valg/lib/libopen-pal.so.0(opal_progress+0x5a)
[0x5324c9a]
[octet:00923] [ 3] /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0 [0x4e59d45]
[octet:00923] [ 4]
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_coll_tuned.so [0x9ccad2a]
[octet:00923] [ 5]
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_coll_tuned.so [0x9cd349e]
[octet:00923] [ 6]
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0(MPI_Barrier+0x72) [0x4e6fa32]
[octet:00923] [ 7] ompi1.3.3-bug(main+0x1ee) [0x409152]
[octet:00923] [ 8] /lib64/libc.so.6(__libc_start_main+0xfd) [0x382501ea2d]
[octet:00923] [ 9] ompi1.3.3-bug [0x408ea9]
[octet:00923] *** End of error message ***
==923==
==923== Process terminating with default action of signal 11 (SIGSEGV)
==923== Access not within mapped region at address 0x40
==923== at 0x8C21009: mca_btl_sm_component_progress (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)
==923== by 0x5324C99: opal_progress (in
/usr/local/openmpi-1.3.3-valg/lib/libopen-pal.so.0.0.0)
==923== by 0x4E59D44: ompi_request_default_wait_all (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==923== by 0x9CCAD29: ompi_coll_tuned_sendrecv_actual (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_coll_tuned.so)
==923== by 0x9CD349D: ompi_coll_tuned_barrier_intra_recursivedoubling
(in /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_coll_tuned.so)
==923== by 0x4E6FA31: PMPI_Barrier (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==923== by 0x409151: main (in /home/research/cary/ompi1.3.3-bug)
==923== If you believe this happened as a result of a stack
==923== overflow in your program's main thread (unlikely but
==923== possible), you can try to increase the size of the
==923== main thread stack using the --main-stacksize= flag.
==923== The main thread stack size used in this run was 10485760.
==923==
==923== HEAP SUMMARY:
==923== in use at exit: 2,062,803 bytes in 2,652 blocks
==923== total heap usage: 10,244 allocs, 7,592 frees, 15,824,029 bytes
allocated
==923==
==923== LEAK SUMMARY:
==923== definitely lost: 1,700 bytes in 29 blocks
==923== indirectly lost: 2,400 bytes in 1 blocks
==923== possibly lost: 2,384 bytes in 22 blocks
==923== still reachable: 2,056,319 bytes in 2,600 blocks
==923== suppressed: 0 bytes in 0 blocks
==923== Rerun with --leak-check=full to see details of leaked memory
==923==
==923== For counts of detected and suppressed errors, rerun with: -v
==923== Use --track-origins=yes to see where uninitialised values come from
==923== ERROR SUMMARY: 5 errors from 5 contexts (suppressed: 7 from 7)
--------------------------------------------------------------------------
mpirun noticed that process rank 7 with PID 923 on node octet.carys.home
exited on signal 11 (Segmentation fault).
--------------------------------------------------------------------------

-- 
Tech-X Corp., 5621 Arapahoe Ave, Suite A, Boulder CO 80303
cary_at_[hidden], p 303-448-0727, f 303-448-7756, NEW CELL 303-881-8572