Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: Re: [OMPI users] Openmpi failure on dual quad linux
From: John R. Cary (cary_at_[hidden])
Date: 2009-10-10 12:20:39


More data:

We have observed this failure on 3 different 8-core, 64-bit boxes, the
previously noted
box plus

benten.caryjr$ uname -a
*** 2.6.9-89.0.7.ELsmp #1 SMP Wed Aug 5 14:08:20 EDT 2009 x86_64 x86_64
x86_64 GNU/Linux
*** 2.6.29.4-167.fc11.x86_64 #1 SMP Wed May 27 17:27:08 EDT 2009 x86_64
x86_64 x86_64 GNU/Linux

It does not appear to occur on a 4-core, 32-bit box:
multipole.cary$ uname -a
*** 2.6.25.14-108.fc9.i686 #1 SMP Mon Aug 4 14:08:11 EDT 2008 i686
athlon i386 GNU/Linux
which has an intermediate kernel.

JC

John R. Cary wrote:
> We have been getting hangs and failures of openmpi-1.3.X
> on an 8-core FC11 box. Details:
>
> Machine:
>
> Linux octet.carys.home 2.6.30.5-43.fc11.x86_64 #1 SMP Thu Aug 27
> 21:39:52 EDT 2009 x86_64 x86_64 x86_64 GNU/Linux
>
> with 2 quad-core cpus.
>
> Compiler:
>
> g++ (GCC) 4.4.1 20090725 (Red Hat 4.4.1-2)
>
> The code is later in this email.
>
>
> OpenMPI configured with
>
> ../configure --prefix=/usr/local/openmpi-1.3.3
> --with-wrapper-ldflags=-Wl,-rpath,/usr/local/openmpi-1.3.3/lib
>
> used to compile and execute:
>
> /usr/local/openmpi-1.3.3/bin/mpicxx ompi1.3.3-bug.cxx -o ompi1.3.3-bug
> /usr/local/openmpi-1.3.3/bin/mpirun -np 8 ompi1.3.3-bug
>
> hangs on various steps: 10, 39, 125. Top shows the code
> using all resources:
>
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+
> COMMAND 5716 cary 20 0 98.6m 4396 2940 R 100.3 0.0
> 0:33.58 ompi1.3.3-bug 5707 cary 20 0 98.7m 4592 3060 R 99.9
> 0.0 0:33.57 ompi1.3.3-bug 5710 cary 20 0 98.6m 4388 2932
> R 99.9 0.0 0:33.61 ompi1.3.3-bug 5713 cary 20 0 98.6m
> 4392 2936 R 99.9 0.0 0:33.61 ompi1.3.3-bug 5719 cary 20
> 0 98.6m 4396 2940 R 99.9 0.0 0:33.60 ompi1.3.3-bug 5722
> cary 20 0 98.6m 4384 2928 R 99.9 0.0 0:33.61
> ompi1.3.3-bug 5728 cary 20 0 98.6m 4408 2952 R 99.9 0.0
> 0:33.53 ompi1.3.3-bug 5725 cary 20 0 98.6m 4404 2948 R
> 99.6 0.0 0:33.57 ompi1.3.3-bug
>
> OpenMPI configured with
>
> ../configure --prefix=/usr/local/openmpi-1.2.9
> --with-wrapper-ldflags=-Wl,-rpath,/usr/local/openmpi-1.2.9/lib
>
> used to compile and execute:
>
> /usr/local/openmpi-1.2.9/bin/mpicxx ompi1.3.3-bug.cxx -o ompi1.3.3-bug
> /usr/local/openmpi-1.2.9/bin/mpirun -np 8 ompi1.3.3-bug
>
> works just fine.
>
>
> So then I started looking for unitialized variables. I rebuilt
> OpenMPI, configured with
>
> ../configure --prefix=/usr/local/openmpi-1.3.3-valg
> --with-valgrind=/contrib/valgrind
> --with-wrapper-ldflags=-Wl,-rpath,/usr/local/openmpi-1.3.3-valg/lib
>
> and used to to compile and execute:
>
> /usr/local/openmpi-1.3.3-valg/bin/mpicxx ompi1.3.3-bug.cxx -o
> ompi1.3.3-bug
> /usr/local/openmpi-1.3.3-valg/bin/mpirun -np 8 valgrind ompi1.3.3-bug
>
>
> and one can get valgrind to segfault, as shown at the very end of this
> email. Other times it hangs on step 1 or another step.
>
> John Cary
>
> THE CODE
>
> $ cat ompi1.3.3-bug.cxx
>
>
> /**
> * A simple test program to demonstrate a problem in OpenMPI 1.3
> */
>
> // mpi includes
> #include <mpi.h>
>
> // std includes
> #include <iostream>
> #include <vector>
>
> // useful hashdefine
> #define ARRAY_SIZE 250
>
> /**
> * Main driver
> */
> int main(int argc, char** argv) {
> // Initialize MPI
> MPI_Init(&argc, &argv);
>
> int rk, sz;
> MPI_Comm_rank(MPI_COMM_WORLD, &rk);
> MPI_Comm_size(MPI_COMM_WORLD, &sz);
>
> // Create some data to pass around
> std::vector<double> d(ARRAY_SIZE);
>
> // Initialize to some values if we aren't rank 0
> if ( rk )
> for ( unsigned i = 0; i < ARRAY_SIZE; ++i )
> d[i] = 2*i + 1;
>
> // Loop until this breaks
> unsigned t = 0;
> while ( 1 ) {
> MPI_Status s;
> if ( rk )
> MPI_Send( &d[0], d.size(), MPI_DOUBLE, 0, 3, MPI_COMM_WORLD );
> else
> for ( int i = 1; i < sz; ++i )
> MPI_Recv( &d[0], d.size(), MPI_DOUBLE, i, 3, MPI_COMM_WORLD, &s );
> MPI_Barrier(MPI_COMM_WORLD);
> std::cout << "Transmission " << ++t << " completed." << std::endl;
> }
>
> // Finalize MPI
> MPI_Finalize();
> }
>
> THE VALGRIND RUN
>
> octet.cary$ /usr/local/openmpi-1.3.3-valg/bin/mpirun -np 8 valgrind
> ompi1.3.3-bug
> ==902== Memcheck, a memory error detector
> ==902== Copyright (C) 2002-2009, and GNU GPL'd, by Julian Seward et al.
> ==902== Using Valgrind-3.5.0 and LibVEX; rerun with -h for copyright info
> ==902== Command: ompi1.3.3-bug
> ==902==
> ==905== Memcheck, a memory error detector
> ==905== Copyright (C) 2002-2009, and GNU GPL'd, by Julian Seward et al.
> ==905== Using Valgrind-3.5.0 and LibVEX; rerun with -h for copyright info
> ==905== Command: ompi1.3.3-bug
> ==905==
> ==908== Memcheck, a memory error detector
> ==908== Copyright (C) 2002-2009, and GNU GPL'd, by Julian Seward et al.
> ==908== Using Valgrind-3.5.0 and LibVEX; rerun with -h for copyright info
> ==908== Command: ompi1.3.3-bug
> ==908==
> ==911== Memcheck, a memory error detector
> ==911== Copyright (C) 2002-2009, and GNU GPL'd, by Julian Seward et al.
> ==911== Using Valgrind-3.5.0 and LibVEX; rerun with -h for copyright info
> ==911== Command: ompi1.3.3-bug
> ==911==
> ==914== Memcheck, a memory error detector
> ==914== Copyright (C) 2002-2009, and GNU GPL'd, by Julian Seward et al.
> ==914== Using Valgrind-3.5.0 and LibVEX; rerun with -h for copyright info
> ==914== Command: ompi1.3.3-bug
> ==914==
> ==917== Memcheck, a memory error detector
> ==917== Copyright (C) 2002-2009, and GNU GPL'd, by Julian Seward et al.
> ==917== Using Valgrind-3.5.0 and LibVEX; rerun with -h for copyright info
> ==917== Command: ompi1.3.3-bug
> ==917==
> ==920== Memcheck, a memory error detector
> ==920== Copyright (C) 2002-2009, and GNU GPL'd, by Julian Seward et al.
> ==920== Using Valgrind-3.5.0 and LibVEX; rerun with -h for copyright info
> ==920== Command: ompi1.3.3-bug
> ==920==
> ==923== Memcheck, a memory error detector
> ==923== Copyright (C) 2002-2009, and GNU GPL'd, by Julian Seward et al.
> ==923== Using Valgrind-3.5.0 and LibVEX; rerun with -h for copyright info
> ==923== Command: ompi1.3.3-bug
> ==923==
> ==917== Warning: client syscall munmap tried to modify addresses
> 0xffffffffffffffff-0xffe
> ==902== Warning: client syscall munmap tried to modify addresses
> 0xffffffffffffffff-0xffe
> ==905== Warning: client syscall munmap tried to modify addresses
> 0xffffffffffffffff-0xffe
> ==914== Warning: client syscall munmap tried to modify addresses
> 0xffffffffffffffff-0xffe
> ==911== Warning: client syscall munmap tried to modify addresses
> 0xffffffffffffffff-0xffe
> ==920== Warning: client syscall munmap tried to modify addresses
> 0xffffffffffffffff-0xffe
> ==923== Warning: client syscall munmap tried to modify addresses
> 0xffffffffffffffff-0xffe
> ==908== Warning: client syscall munmap tried to modify addresses
> 0xffffffffffffffff-0xffe
> ==902== Conditional jump or move depends on uninitialised value(s)
> ==902== at 0x7BDFD64: mca_mpool_sm_alloc (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)
> ==902== by 0x8C1F94D: mca_btl_sm_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)
> ==902== by 0x8816759: mca_bml_r2_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)
> ==902== by 0x8400274: mca_pml_ob1_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)
> ==902== by 0x4E5B048: ompi_mpi_init (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==902== by 0x4E79F65: PMPI_Init (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==902== by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
> ==902==
> ==902== Conditional jump or move depends on uninitialised value(s)
> ==902== at 0x7BDFD64: mca_mpool_sm_alloc (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)
> ==902== by 0x4E3EBDF: ompi_free_list_grow (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==902== by 0x8C1FA22: mca_btl_sm_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)
> ==902== by 0x8816759: mca_bml_r2_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)
> ==902== by 0x8400274: mca_pml_ob1_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)
> ==902== by 0x4E5B048: ompi_mpi_init (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==902== by 0x4E79F65: PMPI_Init (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==902== by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
> ==902==
> ==911== Conditional jump or move depends on uninitialised value(s)
> ==911== at 0x7BDFD64: mca_mpool_sm_alloc (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)
> ==911== by 0x8C1F94D: mca_btl_sm_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)
> ==911== by 0x8816759: mca_bml_r2_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)
> ==911== by 0x8400274: mca_pml_ob1_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)
> ==911== by 0x4E5B048: ompi_mpi_init (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==911== by 0x4E79F65: PMPI_Init (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==911== by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
> ==911==
> ==902== Conditional jump or move depends on uninitialised value(s)
> ==902== at 0x7BDFD64: mca_mpool_sm_alloc (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)
> ==902== by 0x4E3EBDF: ompi_free_list_grow (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==902== by 0x8C1FA93: mca_btl_sm_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)
> ==902== by 0x8816759: mca_bml_r2_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)
> ==902== by 0x8400274: mca_pml_ob1_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)
> ==902== by 0x4E5B048: ompi_mpi_init (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==902== by 0x4E79F65: PMPI_Init (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==914== Conditional jump or move depends on uninitialised value(s)
> ==902== by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
> ==902==
> ==914== at 0x7BDFD64: mca_mpool_sm_alloc (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)
> ==914== by 0x8C1F94D: mca_btl_sm_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)
> ==914== by 0x8816759: mca_bml_r2_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)
> ==914== by 0x8400274: mca_pml_ob1_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)
> ==914== by 0x4E5B048: ompi_mpi_init (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==914== by 0x4E79F65: PMPI_Init (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==914== by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
> ==914==
> ==917== Conditional jump or move depends on uninitialised value(s)
> ==917== at 0x7BDFD64: mca_mpool_sm_alloc (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)
> ==917== by 0x8C1F94D: mca_btl_sm_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)
> ==917== by 0x8816759: mca_bml_r2_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)
> ==917== by 0x8400274: mca_pml_ob1_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)
> ==917== by 0x4E5B048: ompi_mpi_init (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==917== by 0x4E79F65: PMPI_Init (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==917== by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
> ==917==
> ==911== Conditional jump or move depends on uninitialised value(s)
> ==911== at 0x7BDFD64: mca_mpool_sm_alloc (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)
> ==911== by 0x4E3EBDF: ompi_free_list_grow (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==911== by 0x8C1FA22: mca_btl_sm_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)
> ==911== by 0x8816759: mca_bml_r2_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)
> ==911== by 0x8400274: mca_pml_ob1_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)
> ==911== by 0x4E5B048: ompi_mpi_init (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==911== by 0x4E79F65: PMPI_Init (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==911== by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
> ==911==
> ==914== Conditional jump or move depends on uninitialised value(s)
> ==914== at 0x7BDFD64: mca_mpool_sm_alloc (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)
> ==914== by 0x4E3EBDF: ompi_free_list_grow (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==914== by 0x8C1FA22: mca_btl_sm_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)
> ==914== by 0x8816759: mca_bml_r2_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)
> ==914== by 0x8400274: mca_pml_ob1_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)
> ==914== by 0x4E5B048: ompi_mpi_init (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==914== by 0x4E79F65: PMPI_Init (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==914== by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
> ==914==
> ==908== Conditional jump or move depends on uninitialised value(s)
> ==908== at 0x7BDFD64: mca_mpool_sm_alloc (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)
> ==908== by 0x8C1F94D: mca_btl_sm_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)
> ==908== by 0x8816759: mca_bml_r2_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)
> ==908== by 0x8400274: mca_pml_ob1_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)
> ==908== by 0x4E5B048: ompi_mpi_init (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==908== by 0x4E79F65: PMPI_Init (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==908== by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
> ==908==
> ==902== Conditional jump or move depends on uninitialised value(s)
> ==902== at 0x7BDFD64: mca_mpool_sm_alloc (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)
> ==902== by 0x8C1F0FC: mca_btl_sm_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)
> ==902== by 0x8816759: mca_bml_r2_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)
> ==902== by 0x8400274: mca_pml_ob1_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)
> ==902== by 0x4E5B048: ompi_mpi_init (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==902== by 0x4E79F65: PMPI_Init (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==902== by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
> ==902==
> ==911== Conditional jump or move depends on uninitialised value(s)
> ==911== at 0x7BDFD64: mca_mpool_sm_alloc (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)
> ==911== by 0x4E3EBDF: ompi_free_list_grow (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==923== Conditional jump or move depends on uninitialised value(s)
> ==911== by 0x8C1FA93: mca_btl_sm_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)
> ==911== by 0x8816759: mca_bml_r2_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)
> ==911== by 0x8400274: mca_pml_ob1_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)
> ==911== by 0x4E5B048: ompi_mpi_init (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==911== by 0x4E79F65: PMPI_Init (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==911== by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
> ==911==
> ==923== at 0x7BDFD64: mca_mpool_sm_alloc (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)
> ==923== by 0x8C1F94D: mca_btl_sm_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)
> ==920== Conditional jump or move depends on uninitialised value(s)
> ==923== by 0x8816759: mca_bml_r2_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)
> ==923== by 0x8400274: mca_pml_ob1_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)
> ==923== by 0x4E5B048: ompi_mpi_init (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==923== by 0x4E79F65: PMPI_Init (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==923== by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
> ==923==
> ==920== at 0x7BDFD64: mca_mpool_sm_alloc (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)
> ==920== by 0x8C1F94D: mca_btl_sm_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)
> ==920== by 0x8816759: mca_bml_r2_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)
> ==920== by 0x8400274: mca_pml_ob1_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)
> ==920== by 0x4E5B048: ompi_mpi_init (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==920== by 0x4E79F65: PMPI_Init (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==920== by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
> ==920==
> ==914== Conditional jump or move depends on uninitialised value(s)
> ==914== at 0x7BDFD64: mca_mpool_sm_alloc (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)
> ==914== by 0x4E3EBDF: ompi_free_list_grow (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==914== by 0x8C1FA93: mca_btl_sm_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)
> ==914== by 0x8816759: mca_bml_r2_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)
> ==914== by 0x8400274: mca_pml_ob1_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)
> ==914== by 0x4E5B048: ompi_mpi_init (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==914== by 0x4E79F65: PMPI_Init (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==914== by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
> ==914==
> ==917== Conditional jump or move depends on uninitialised value(s)
> ==917== at 0x7BDFD64: mca_mpool_sm_alloc (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)
> ==917== by 0x4E3EBDF: ompi_free_list_grow (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==917== by 0x8C1FA22: mca_btl_sm_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)
> ==917== by 0x8816759: mca_bml_r2_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)
> ==917== by 0x8400274: mca_pml_ob1_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)
> ==917== by 0x4E5B048: ompi_mpi_init (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==917== by 0x4E79F65: PMPI_Init (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==917== by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
> ==917==
> ==917== Conditional jump or move depends on uninitialised value(s)
> ==908== Conditional jump or move depends on uninitialised value(s)
> ==908== at 0x7BDFD64: mca_mpool_sm_alloc (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)
> ==908== by 0x4E3EBDF: ompi_free_list_grow (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==908== by 0x8C1FA22: mca_btl_sm_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)
> ==908== by 0x8816759: mca_bml_r2_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)
> ==908== by 0x8400274: mca_pml_ob1_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)
> ==908== by 0x4E5B048: ompi_mpi_init (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==908== by 0x4E79F65: PMPI_Init (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==908== by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
> ==908==
> ==917== at 0x7BDFD64: mca_mpool_sm_alloc (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)
> ==917== by 0x4E3EBDF: ompi_free_list_grow (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==917== by 0x8C1FA93: mca_btl_sm_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)
> ==917== by 0x8816759: mca_bml_r2_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)
> ==917== by 0x8400274: mca_pml_ob1_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)
> ==917== by 0x4E5B048: ompi_mpi_init (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==911== Conditional jump or move depends on uninitialised value(s)
> ==911== at 0x7BDFD64: mca_mpool_sm_alloc (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)
> ==911== by 0x8C1F0FC: mca_btl_sm_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)
> ==911== by 0x8816759: mca_bml_r2_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)
> ==911== by 0x8400274: mca_pml_ob1_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)
> ==911== by 0x4E5B048: ompi_mpi_init (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==911== by 0x4E79F65: PMPI_Init (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==911== by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
> ==911==
> ==917== by 0x4E79F65: PMPI_Init (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==917== by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
> ==914== Conditional jump or move depends on uninitialised value(s)
> ==914== at 0x7BDFD64: mca_mpool_sm_alloc (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)
> ==914== by 0x8C1F0FC: mca_btl_sm_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)
> ==914== by 0x8816759: mca_bml_r2_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)
> ==914== by 0x8400274: mca_pml_ob1_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)
> ==914== by 0x4E5B048: ompi_mpi_init (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==914== by 0x4E79F65: PMPI_Init (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==914== by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
> ==914==
> ==917==
> ==923== Conditional jump or move depends on uninitialised value(s)
> ==923== at 0x7BDFD64: mca_mpool_sm_alloc (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)
> ==923== by 0x4E3EBDF: ompi_free_list_grow (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==923== by 0x8C1FA22: mca_btl_sm_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)
> ==923== by 0x8816759: mca_bml_r2_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)
> ==923== by 0x8400274: mca_pml_ob1_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)
> ==923== by 0x4E5B048: ompi_mpi_init (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==923== by 0x4E79F65: PMPI_Init (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==923== by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
> ==923==
> ==920== Conditional jump or move depends on uninitialised value(s)
> ==920== at 0x7BDFD64: mca_mpool_sm_alloc (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)
> ==920== by 0x4E3EBDF: ompi_free_list_grow (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==920== by 0x8C1FA22: mca_btl_sm_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)
> ==920== by 0x8816759: mca_bml_r2_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)
> ==920== by 0x8400274: mca_pml_ob1_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)
> ==920== by 0x4E5B048: ompi_mpi_init (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==920== by 0x4E79F65: PMPI_Init (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==920== by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
> ==920==
> ==908== Conditional jump or move depends on uninitialised value(s)
> ==908== at 0x7BDFD64: mca_mpool_sm_alloc (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)
> ==908== by 0x4E3EBDF: ompi_free_list_grow (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==908== by 0x8C1FA93: mca_btl_sm_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)
> ==908== by 0x8816759: mca_bml_r2_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)
> ==908== by 0x8400274: mca_pml_ob1_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)
> ==908== by 0x4E5B048: ompi_mpi_init (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==908== by 0x4E79F65: PMPI_Init (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==908== by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
> ==908==
> ==923== Conditional jump or move depends on uninitialised value(s)
> ==923== at 0x7BDFD64: mca_mpool_sm_alloc (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)
> ==923== by 0x4E3EBDF: ompi_free_list_grow (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==920== Conditional jump or move depends on uninitialised value(s)
> ==920== at 0x7BDFD64: mca_mpool_sm_alloc (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)
> ==920== by 0x4E3EBDF: ompi_free_list_grow (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==920== by 0x8C1FA93: mca_btl_sm_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)
> ==923== by 0x8C1FA93: mca_btl_sm_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)
> ==923== by 0x8816759: mca_bml_r2_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)
> ==923== by 0x8400274: mca_pml_ob1_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)
> ==923== by 0x4E5B048: ompi_mpi_init (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==923== by 0x4E79F65: PMPI_Init (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==923== by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
> ==923==
> ==920== by 0x8816759: mca_bml_r2_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)
> ==920== by 0x8400274: mca_pml_ob1_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)
> ==920== by 0x4E5B048: ompi_mpi_init (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==920== by 0x4E79F65: PMPI_Init (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==920== by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
> ==920==
> ==905== Conditional jump or move depends on uninitialised value(s)
> ==905== at 0x7BDFD64: mca_mpool_sm_alloc (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)
> ==905== by 0x8C1F94D: mca_btl_sm_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)
> ==905== by 0x8816759: mca_bml_r2_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)
> ==905== by 0x8400274: mca_pml_ob1_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)
> ==905== by 0x4E5B048: ompi_mpi_init (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==905== by 0x4E79F65: PMPI_Init (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==905== by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
> ==905==
> ==917== Conditional jump or move depends on uninitialised value(s)
> ==917== at 0x7BDFD64: mca_mpool_sm_alloc (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)
> ==917== by 0x8C1F0FC: mca_btl_sm_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)
> ==917== by 0x8816759: mca_bml_r2_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)
> ==917== by 0x8400274: mca_pml_ob1_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)
> ==917== by 0x4E5B048: ompi_mpi_init (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==917== by 0x4E79F65: PMPI_Init (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==917== by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
> ==917==
> ==908== Conditional jump or move depends on uninitialised value(s)
> ==908== at 0x7BDFD64: mca_mpool_sm_alloc (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)
> ==908== by 0x8C1F0FC: mca_btl_sm_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)
> ==908== by 0x8816759: mca_bml_r2_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)
> ==908== by 0x8400274: mca_pml_ob1_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)
> ==908== by 0x4E5B048: ompi_mpi_init (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==908== by 0x4E79F65: PMPI_Init (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==908== by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
> ==908==
> ==905== Conditional jump or move depends on uninitialised value(s)
> ==905== at 0x7BDFD64: mca_mpool_sm_alloc (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)
> ==905== by 0x4E3EBDF: ompi_free_list_grow (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==905== by 0x8C1FA22: mca_btl_sm_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)
> ==905== by 0x8816759: mca_bml_r2_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)
> ==905== by 0x8400274: mca_pml_ob1_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)
> ==905== by 0x4E5B048: ompi_mpi_init (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==905== by 0x4E79F65: PMPI_Init (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==905== by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
> ==905==
> ==923== Conditional jump or move depends on uninitialised value(s)
> ==923== at 0x7BDFD64: mca_mpool_sm_alloc (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)
> ==923== by 0x8C1F0FC: mca_btl_sm_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)
> ==923== by 0x8816759: mca_bml_r2_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)
> ==923== by 0x8400274: mca_pml_ob1_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)
> ==923== by 0x4E5B048: ompi_mpi_init (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==923== by 0x4E79F65: PMPI_Init (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==923== by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
> ==923==
> ==920== Conditional jump or move depends on uninitialised value(s)
> ==920== at 0x7BDFD64: mca_mpool_sm_alloc (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)
> ==920== by 0x8C1F0FC: mca_btl_sm_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)
> ==920== by 0x8816759: mca_bml_r2_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)
> ==920== by 0x8400274: mca_pml_ob1_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)
> ==920== by 0x4E5B048: ompi_mpi_init (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==920== by 0x4E79F65: PMPI_Init (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==920== by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
> ==920==
> ==905== Conditional jump or move depends on uninitialised value(s)
> ==905== at 0x7BDFD64: mca_mpool_sm_alloc (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)
> ==905== by 0x4E3EBDF: ompi_free_list_grow (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==905== by 0x8C1FA93: mca_btl_sm_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)
> ==905== by 0x8816759: mca_bml_r2_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)
> ==905== by 0x8400274: mca_pml_ob1_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)
> ==905== by 0x4E5B048: ompi_mpi_init (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==905== by 0x4E79F65: PMPI_Init (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==905== by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
> ==905==
> ==917== Conditional jump or move depends on uninitialised value(s)
> ==917== at 0x7BDFD64: mca_mpool_sm_alloc (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)
> ==917== by 0x4E3EBDF: ompi_free_list_grow (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==917== by 0x4E3EC5F: ompi_free_list_resize (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==917== by 0x8C1F630: mca_btl_sm_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)
> ==917== by 0x8816759: mca_bml_r2_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)
> ==917== by 0x8400274: mca_pml_ob1_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)
> ==917== by 0x4E5B048: ompi_mpi_init (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==917== by 0x4E79F65: PMPI_Init (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==917== by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
> ==917==
> ==908== Conditional jump or move depends on uninitialised value(s)
> ==908== at 0x7BDFD64: mca_mpool_sm_alloc (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)
> ==908== by 0x4E3EBDF: ompi_free_list_grow (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==908== by 0x4E3EC5F: ompi_free_list_resize (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==908== by 0x8C1F630: mca_btl_sm_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)
> ==908== by 0x8816759: mca_bml_r2_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)
> ==908== by 0x8400274: mca_pml_ob1_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)
> ==908== by 0x4E5B048: ompi_mpi_init (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==908== by 0x4E79F65: PMPI_Init (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==908== by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
> ==908==
> ==923== Conditional jump or move depends on uninitialised value(s)
> ==923== at 0x7BDFD64: mca_mpool_sm_alloc (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)
> ==923== by 0x4E3EBDF: ompi_free_list_grow (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==923== by 0x4E3EC5F: ompi_free_list_resize (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==923== by 0x8C1F630: mca_btl_sm_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)
> ==923== by 0x8816759: mca_bml_r2_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)
> ==923== by 0x8400274: mca_pml_ob1_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)
> ==923== by 0x4E5B048: ompi_mpi_init (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==923== by 0x4E79F65: PMPI_Init (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==923== by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
> ==923==
> ==920== Conditional jump or move depends on uninitialised value(s)
> ==920== at 0x7BDFD64: mca_mpool_sm_alloc (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)
> ==920== by 0x4E3EBDF: ompi_free_list_grow (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==920== by 0x4E3EC5F: ompi_free_list_resize (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==920== by 0x8C1F630: mca_btl_sm_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)
> ==920== by 0x8816759: mca_bml_r2_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)
> ==920== by 0x8400274: mca_pml_ob1_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)
> ==920== by 0x4E5B048: ompi_mpi_init (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==920== by 0x4E79F65: PMPI_Init (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==920== by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
> ==920==
> ==905== Conditional jump or move depends on uninitialised value(s)
> ==905== at 0x7BDFD64: mca_mpool_sm_alloc (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)
> ==905== by 0x8C1F0FC: mca_btl_sm_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)
> ==905== by 0x8816759: mca_bml_r2_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)
> ==905== by 0x8400274: mca_pml_ob1_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)
> ==905== by 0x4E5B048: ompi_mpi_init (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==905== by 0x4E79F65: PMPI_Init (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==905== by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
> ==905==
> ==905== Conditional jump or move depends on uninitialised value(s)
> ==905== at 0x7BDFD64: mca_mpool_sm_alloc (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)
> ==905== by 0x4E3EBDF: ompi_free_list_grow (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==905== by 0x4E3EC5F: ompi_free_list_resize (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==905== by 0x8C1F630: mca_btl_sm_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)
> ==905== by 0x8816759: mca_bml_r2_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)
> ==905== by 0x8400274: mca_pml_ob1_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)
> ==905== by 0x4E5B048: ompi_mpi_init (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==905== by 0x4E79F65: PMPI_Init (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==905== by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
> ==905==
> ==902== Conditional jump or move depends on uninitialised value(s)
> ==902== at 0x7BDFD64: mca_mpool_sm_alloc (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)
> ==902== by 0x4E3EBDF: ompi_free_list_grow (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==902== by 0x4E3EC5F: ompi_free_list_resize (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==902== by 0x8C1F630: mca_btl_sm_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)
> ==902== by 0x8816759: mca_bml_r2_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)
> ==902== by 0x8400274: mca_pml_ob1_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)
> ==902== by 0x4E5B048: ompi_mpi_init (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==902== by 0x4E79F65: PMPI_Init (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==902== by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
> ==902==
> ==911== Conditional jump or move depends on uninitialised value(s)
> ==911== at 0x7BDFD64: mca_mpool_sm_alloc (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)
> ==911== by 0x4E3EBDF: ompi_free_list_grow (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==911== by 0x4E3EC5F: ompi_free_list_resize (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==911== by 0x8C1F630: mca_btl_sm_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)
> ==911== by 0x8816759: mca_bml_r2_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)
> ==911== by 0x8400274: mca_pml_ob1_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)
> ==911== by 0x4E5B048: ompi_mpi_init (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==911== by 0x4E79F65: PMPI_Init (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==911== by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
> ==911==
> ==914== Conditional jump or move depends on uninitialised value(s)
> ==914== at 0x7BDFD64: mca_mpool_sm_alloc (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)
> ==914== by 0x4E3EBDF: ompi_free_list_grow (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==914== by 0x4E3EC5F: ompi_free_list_resize (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==914== by 0x8C1F630: mca_btl_sm_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)
> ==914== by 0x8816759: mca_bml_r2_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)
> ==914== by 0x8400274: mca_pml_ob1_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)
> ==914== by 0x4E5B048: ompi_mpi_init (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==914== by 0x4E79F65: PMPI_Init (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==914== by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
> ==914==
> [octet:00923] *** Process received signal ***
> [octet:00923] Signal: Segmentation fault (11)
> [octet:00923] Signal code: Address not mapped (1)
> [octet:00923] Failing at address: 0x40
> [octet:00923] [ 0] /lib64/libpthread.so.0 [0x3825c0eee0]
> [octet:00923] [ 1]
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so [0x8c21009]
> [octet:00923] [ 2]
> /usr/local/openmpi-1.3.3-valg/lib/libopen-pal.so.0(opal_progress+0x5a)
> [0x5324c9a]
> [octet:00923] [ 3] /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0
> [0x4e59d45]
> [octet:00923] [ 4]
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_coll_tuned.so [0x9ccad2a]
> [octet:00923] [ 5]
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_coll_tuned.so [0x9cd349e]
> [octet:00923] [ 6]
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0(MPI_Barrier+0x72)
> [0x4e6fa32]
> [octet:00923] [ 7] ompi1.3.3-bug(main+0x1ee) [0x409152]
> [octet:00923] [ 8] /lib64/libc.so.6(__libc_start_main+0xfd)
> [0x382501ea2d]
> [octet:00923] [ 9] ompi1.3.3-bug [0x408ea9]
> [octet:00923] *** End of error message ***
> ==923==
> ==923== Process terminating with default action of signal 11 (SIGSEGV)
> ==923== Access not within mapped region at address 0x40
> ==923== at 0x8C21009: mca_btl_sm_component_progress (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)
> ==923== by 0x5324C99: opal_progress (in
> /usr/local/openmpi-1.3.3-valg/lib/libopen-pal.so.0.0.0)
> ==923== by 0x4E59D44: ompi_request_default_wait_all (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==923== by 0x9CCAD29: ompi_coll_tuned_sendrecv_actual (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_coll_tuned.so)
> ==923== by 0x9CD349D:
> ompi_coll_tuned_barrier_intra_recursivedoubling (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_coll_tuned.so)
> ==923== by 0x4E6FA31: PMPI_Barrier (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==923== by 0x409151: main (in /home/research/cary/ompi1.3.3-bug)
> ==923== If you believe this happened as a result of a stack
> ==923== overflow in your program's main thread (unlikely but
> ==923== possible), you can try to increase the size of the
> ==923== main thread stack using the --main-stacksize= flag.
> ==923== The main thread stack size used in this run was 10485760.
> ==923==
> ==923== HEAP SUMMARY:
> ==923== in use at exit: 2,062,803 bytes in 2,652 blocks
> ==923== total heap usage: 10,244 allocs, 7,592 frees, 15,824,029
> bytes allocated
> ==923==
> ==923== LEAK SUMMARY:
> ==923== definitely lost: 1,700 bytes in 29 blocks
> ==923== indirectly lost: 2,400 bytes in 1 blocks
> ==923== possibly lost: 2,384 bytes in 22 blocks
> ==923== still reachable: 2,056,319 bytes in 2,600 blocks
> ==923== suppressed: 0 bytes in 0 blocks
> ==923== Rerun with --leak-check=full to see details of leaked memory
> ==923==
> ==923== For counts of detected and suppressed errors, rerun with: -v
> ==923== Use --track-origins=yes to see where uninitialised values come
> from
> ==923== ERROR SUMMARY: 5 errors from 5 contexts (suppressed: 7 from 7)
> --------------------------------------------------------------------------
>
> mpirun noticed that process rank 7 with PID 923 on node
> octet.carys.home exited on signal 11 (Segmentation fault).
> --------------------------------------------------------------------------
>
>
>

-- 
Tech-X Corp., 5621 Arapahoe Ave, Suite A, Boulder CO 80303
cary_at_[hidden], p 303-448-0727, f 303-448-7756, NEW CELL 303-881-8572