Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |  

This web mail archive is frozen.

This page is part of a frozen web archive of this mailing list.

You can still navigate around this archive, but know that no new mails have been added to it since July of 2016.

Click here to be taken to the new web archives of this list; it includes all the mails that are in this frozen archive plus all new mails that have been sent to the list since it was migrated to the new archives.

Subject: Re: [OMPI users] Openmpi failure on dual quad linux
From: John R. Cary (cary_at_[hidden])
Date: 2009-10-10 12:20:39


More data:

We have observed this failure on 3 different 8-core, 64-bit boxes, the
previously noted
box plus

benten.caryjr$ uname -a
*** 2.6.9-89.0.7.ELsmp #1 SMP Wed Aug 5 14:08:20 EDT 2009 x86_64 x86_64
x86_64 GNU/Linux
*** 2.6.29.4-167.fc11.x86_64 #1 SMP Wed May 27 17:27:08 EDT 2009 x86_64
x86_64 x86_64 GNU/Linux

It does not appear to occur on a 4-core, 32-bit box:
multipole.cary$ uname -a
*** 2.6.25.14-108.fc9.i686 #1 SMP Mon Aug 4 14:08:11 EDT 2008 i686
athlon i386 GNU/Linux
which has an intermediate kernel.

JC

John R. Cary wrote:
> We have been getting hangs and failures of openmpi-1.3.X
> on an 8-core FC11 box. Details:
>
> Machine:
>
> Linux octet.carys.home 2.6.30.5-43.fc11.x86_64 #1 SMP Thu Aug 27
> 21:39:52 EDT 2009 x86_64 x86_64 x86_64 GNU/Linux
>
> with 2 quad-core cpus.
>
> Compiler:
>
> g++ (GCC) 4.4.1 20090725 (Red Hat 4.4.1-2)
>
> The code is later in this email.
>
>
> OpenMPI configured with
>
> ../configure --prefix=/usr/local/openmpi-1.3.3
> --with-wrapper-ldflags=-Wl,-rpath,/usr/local/openmpi-1.3.3/lib
>
> used to compile and execute:
>
> /usr/local/openmpi-1.3.3/bin/mpicxx ompi1.3.3-bug.cxx -o ompi1.3.3-bug
> /usr/local/openmpi-1.3.3/bin/mpirun -np 8 ompi1.3.3-bug
>
> hangs on various steps: 10, 39, 125. Top shows the code
> using all resources:
>
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+
> COMMAND 5716 cary 20 0 98.6m 4396 2940 R 100.3 0.0
> 0:33.58 ompi1.3.3-bug 5707 cary 20 0 98.7m 4592 3060 R 99.9
> 0.0 0:33.57 ompi1.3.3-bug 5710 cary 20 0 98.6m 4388 2932
> R 99.9 0.0 0:33.61 ompi1.3.3-bug 5713 cary 20 0 98.6m
> 4392 2936 R 99.9 0.0 0:33.61 ompi1.3.3-bug 5719 cary 20
> 0 98.6m 4396 2940 R 99.9 0.0 0:33.60 ompi1.3.3-bug 5722
> cary 20 0 98.6m 4384 2928 R 99.9 0.0 0:33.61
> ompi1.3.3-bug 5728 cary 20 0 98.6m 4408 2952 R 99.9 0.0
> 0:33.53 ompi1.3.3-bug 5725 cary 20 0 98.6m 4404 2948 R
> 99.6 0.0 0:33.57 ompi1.3.3-bug
>
> OpenMPI configured with
>
> ../configure --prefix=/usr/local/openmpi-1.2.9
> --with-wrapper-ldflags=-Wl,-rpath,/usr/local/openmpi-1.2.9/lib
>
> used to compile and execute:
>
> /usr/local/openmpi-1.2.9/bin/mpicxx ompi1.3.3-bug.cxx -o ompi1.3.3-bug
> /usr/local/openmpi-1.2.9/bin/mpirun -np 8 ompi1.3.3-bug
>
> works just fine.
>
>
> So then I started looking for unitialized variables. I rebuilt
> OpenMPI, configured with
>
> ../configure --prefix=/usr/local/openmpi-1.3.3-valg
> --with-valgrind=/contrib/valgrind
> --with-wrapper-ldflags=-Wl,-rpath,/usr/local/openmpi-1.3.3-valg/lib
>
> and used to to compile and execute:
>
> /usr/local/openmpi-1.3.3-valg/bin/mpicxx ompi1.3.3-bug.cxx -o
> ompi1.3.3-bug
> /usr/local/openmpi-1.3.3-valg/bin/mpirun -np 8 valgrind ompi1.3.3-bug
>
>
> and one can get valgrind to segfault, as shown at the very end of this
> email. Other times it hangs on step 1 or another step.
>
> John Cary
>
> THE CODE
>
> $ cat ompi1.3.3-bug.cxx
>
>
> /**
> * A simple test program to demonstrate a problem in OpenMPI 1.3
> */
>
> // mpi includes
> #include <mpi.h>
>
> // std includes
> #include <iostream>
> #include <vector>
>
> // useful hashdefine
> #define ARRAY_SIZE 250
>
> /**
> * Main driver
> */
> int main(int argc, char** argv) {
> // Initialize MPI
> MPI_Init(&argc, &argv);
>
> int rk, sz;
> MPI_Comm_rank(MPI_COMM_WORLD, &rk);
> MPI_Comm_size(MPI_COMM_WORLD, &sz);
>
> // Create some data to pass around
> std::vector<double> d(ARRAY_SIZE);
>
> // Initialize to some values if we aren't rank 0
> if ( rk )
> for ( unsigned i = 0; i < ARRAY_SIZE; ++i )
> d[i] = 2*i + 1;
>
> // Loop until this breaks
> unsigned t = 0;
> while ( 1 ) {
> MPI_Status s;
> if ( rk )
> MPI_Send( &d[0], d.size(), MPI_DOUBLE, 0, 3, MPI_COMM_WORLD );
> else
> for ( int i = 1; i < sz; ++i )
> MPI_Recv( &d[0], d.size(), MPI_DOUBLE, i, 3, MPI_COMM_WORLD, &s );
> MPI_Barrier(MPI_COMM_WORLD);
> std::cout << "Transmission " << ++t << " completed." << std::endl;
> }
>
> // Finalize MPI
> MPI_Finalize();
> }
>
> THE VALGRIND RUN
>
> octet.cary$ /usr/local/openmpi-1.3.3-valg/bin/mpirun -np 8 valgrind
> ompi1.3.3-bug
> ==902== Memcheck, a memory error detector
> ==902== Copyright (C) 2002-2009, and GNU GPL'd, by Julian Seward et al.
> ==902== Using Valgrind-3.5.0 and LibVEX; rerun with -h for copyright info
> ==902== Command: ompi1.3.3-bug
> ==902==
> ==905== Memcheck, a memory error detector
> ==905== Copyright (C) 2002-2009, and GNU GPL'd, by Julian Seward et al.
> ==905== Using Valgrind-3.5.0 and LibVEX; rerun with -h for copyright info
> ==905== Command: ompi1.3.3-bug
> ==905==
> ==908== Memcheck, a memory error detector
> ==908== Copyright (C) 2002-2009, and GNU GPL'd, by Julian Seward et al.
> ==908== Using Valgrind-3.5.0 and LibVEX; rerun with -h for copyright info
> ==908== Command: ompi1.3.3-bug
> ==908==
> ==911== Memcheck, a memory error detector
> ==911== Copyright (C) 2002-2009, and GNU GPL'd, by Julian Seward et al.
> ==911== Using Valgrind-3.5.0 and LibVEX; rerun with -h for copyright info
> ==911== Command: ompi1.3.3-bug
> ==911==
> ==914== Memcheck, a memory error detector
> ==914== Copyright (C) 2002-2009, and GNU GPL'd, by Julian Seward et al.
> ==914== Using Valgrind-3.5.0 and LibVEX; rerun with -h for copyright info
> ==914== Command: ompi1.3.3-bug
> ==914==
> ==917== Memcheck, a memory error detector
> ==917== Copyright (C) 2002-2009, and GNU GPL'd, by Julian Seward et al.
> ==917== Using Valgrind-3.5.0 and LibVEX; rerun with -h for copyright info
> ==917== Command: ompi1.3.3-bug
> ==917==
> ==920== Memcheck, a memory error detector
> ==920== Copyright (C) 2002-2009, and GNU GPL'd, by Julian Seward et al.
> ==920== Using Valgrind-3.5.0 and LibVEX; rerun with -h for copyright info
> ==920== Command: ompi1.3.3-bug
> ==920==
> ==923== Memcheck, a memory error detector
> ==923== Copyright (C) 2002-2009, and GNU GPL'd, by Julian Seward et al.
> ==923== Using Valgrind-3.5.0 and LibVEX; rerun with -h for copyright info
> ==923== Command: ompi1.3.3-bug
> ==923==
> ==917== Warning: client syscall munmap tried to modify addresses
> 0xffffffffffffffff-0xffe
> ==902== Warning: client syscall munmap tried to modify addresses
> 0xffffffffffffffff-0xffe
> ==905== Warning: client syscall munmap tried to modify addresses
> 0xffffffffffffffff-0xffe
> ==914== Warning: client syscall munmap tried to modify addresses
> 0xffffffffffffffff-0xffe
> ==911== Warning: client syscall munmap tried to modify addresses
> 0xffffffffffffffff-0xffe
> ==920== Warning: client syscall munmap tried to modify addresses
> 0xffffffffffffffff-0xffe
> ==923== Warning: client syscall munmap tried to modify addresses
> 0xffffffffffffffff-0xffe
> ==908== Warning: client syscall munmap tried to modify addresses
> 0xffffffffffffffff-0xffe
> ==902== Conditional jump or move depends on uninitialised value(s)
> ==902== at 0x7BDFD64: mca_mpool_sm_alloc (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)
> ==902== by 0x8C1F94D: mca_btl_sm_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)
> ==902== by 0x8816759: mca_bml_r2_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)
> ==902== by 0x8400274: mca_pml_ob1_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)
> ==902== by 0x4E5B048: ompi_mpi_init (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==902== by 0x4E79F65: PMPI_Init (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==902== by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
> ==902==
> ==902== Conditional jump or move depends on uninitialised value(s)
> ==902== at 0x7BDFD64: mca_mpool_sm_alloc (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)
> ==902== by 0x4E3EBDF: ompi_free_list_grow (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==902== by 0x8C1FA22: mca_btl_sm_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)
> ==902== by 0x8816759: mca_bml_r2_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)
> ==902== by 0x8400274: mca_pml_ob1_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)
> ==902== by 0x4E5B048: ompi_mpi_init (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==902== by 0x4E79F65: PMPI_Init (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==902== by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
> ==902==
> ==911== Conditional jump or move depends on uninitialised value(s)
> ==911== at 0x7BDFD64: mca_mpool_sm_alloc (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)
> ==911== by 0x8C1F94D: mca_btl_sm_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)
> ==911== by 0x8816759: mca_bml_r2_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)
> ==911== by 0x8400274: mca_pml_ob1_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)
> ==911== by 0x4E5B048: ompi_mpi_init (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==911== by 0x4E79F65: PMPI_Init (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==911== by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
> ==911==
> ==902== Conditional jump or move depends on uninitialised value(s)
> ==902== at 0x7BDFD64: mca_mpool_sm_alloc (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)
> ==902== by 0x4E3EBDF: ompi_free_list_grow (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==902== by 0x8C1FA93: mca_btl_sm_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)
> ==902== by 0x8816759: mca_bml_r2_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)
> ==902== by 0x8400274: mca_pml_ob1_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)
> ==902== by 0x4E5B048: ompi_mpi_init (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==902== by 0x4E79F65: PMPI_Init (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==914== Conditional jump or move depends on uninitialised value(s)
> ==902== by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
> ==902==
> ==914== at 0x7BDFD64: mca_mpool_sm_alloc (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)
> ==914== by 0x8C1F94D: mca_btl_sm_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)
> ==914== by 0x8816759: mca_bml_r2_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)
> ==914== by 0x8400274: mca_pml_ob1_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)
> ==914== by 0x4E5B048: ompi_mpi_init (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==914== by 0x4E79F65: PMPI_Init (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==914== by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
> ==914==
> ==917== Conditional jump or move depends on uninitialised value(s)
> ==917== at 0x7BDFD64: mca_mpool_sm_alloc (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)
> ==917== by 0x8C1F94D: mca_btl_sm_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)
> ==917== by 0x8816759: mca_bml_r2_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)
> ==917== by 0x8400274: mca_pml_ob1_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)
> ==917== by 0x4E5B048: ompi_mpi_init (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==917== by 0x4E79F65: PMPI_Init (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==917== by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
> ==917==
> ==911== Conditional jump or move depends on uninitialised value(s)
> ==911== at 0x7BDFD64: mca_mpool_sm_alloc (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)
> ==911== by 0x4E3EBDF: ompi_free_list_grow (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==911== by 0x8C1FA22: mca_btl_sm_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)
> ==911== by 0x8816759: mca_bml_r2_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)
> ==911== by 0x8400274: mca_pml_ob1_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)
> ==911== by 0x4E5B048: ompi_mpi_init (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==911== by 0x4E79F65: PMPI_Init (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==911== by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
> ==911==
> ==914== Conditional jump or move depends on uninitialised value(s)
> ==914== at 0x7BDFD64: mca_mpool_sm_alloc (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)
> ==914== by 0x4E3EBDF: ompi_free_list_grow (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==914== by 0x8C1FA22: mca_btl_sm_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)
> ==914== by 0x8816759: mca_bml_r2_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)
> ==914== by 0x8400274: mca_pml_ob1_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)
> ==914== by 0x4E5B048: ompi_mpi_init (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==914== by 0x4E79F65: PMPI_Init (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==914== by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
> ==914==
> ==908== Conditional jump or move depends on uninitialised value(s)
> ==908== at 0x7BDFD64: mca_mpool_sm_alloc (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)
> ==908== by 0x8C1F94D: mca_btl_sm_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)
> ==908== by 0x8816759: mca_bml_r2_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)
> ==908== by 0x8400274: mca_pml_ob1_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)
> ==908== by 0x4E5B048: ompi_mpi_init (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==908== by 0x4E79F65: PMPI_Init (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==908== by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
> ==908==
> ==902== Conditional jump or move depends on uninitialised value(s)
> ==902== at 0x7BDFD64: mca_mpool_sm_alloc (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)
> ==902== by 0x8C1F0FC: mca_btl_sm_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)
> ==902== by 0x8816759: mca_bml_r2_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)
> ==902== by 0x8400274: mca_pml_ob1_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)
> ==902== by 0x4E5B048: ompi_mpi_init (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==902== by 0x4E79F65: PMPI_Init (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==902== by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
> ==902==
> ==911== Conditional jump or move depends on uninitialised value(s)
> ==911== at 0x7BDFD64: mca_mpool_sm_alloc (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)
> ==911== by 0x4E3EBDF: ompi_free_list_grow (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==923== Conditional jump or move depends on uninitialised value(s)
> ==911== by 0x8C1FA93: mca_btl_sm_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)
> ==911== by 0x8816759: mca_bml_r2_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)
> ==911== by 0x8400274: mca_pml_ob1_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)
> ==911== by 0x4E5B048: ompi_mpi_init (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==911== by 0x4E79F65: PMPI_Init (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==911== by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
> ==911==
> ==923== at 0x7BDFD64: mca_mpool_sm_alloc (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)
> ==923== by 0x8C1F94D: mca_btl_sm_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)
> ==920== Conditional jump or move depends on uninitialised value(s)
> ==923== by 0x8816759: mca_bml_r2_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)
> ==923== by 0x8400274: mca_pml_ob1_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)
> ==923== by 0x4E5B048: ompi_mpi_init (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==923== by 0x4E79F65: PMPI_Init (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==923== by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
> ==923==
> ==920== at 0x7BDFD64: mca_mpool_sm_alloc (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)
> ==920== by 0x8C1F94D: mca_btl_sm_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)
> ==920== by 0x8816759: mca_bml_r2_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)
> ==920== by 0x8400274: mca_pml_ob1_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)
> ==920== by 0x4E5B048: ompi_mpi_init (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==920== by 0x4E79F65: PMPI_Init (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==920== by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
> ==920==
> ==914== Conditional jump or move depends on uninitialised value(s)
> ==914== at 0x7BDFD64: mca_mpool_sm_alloc (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)
> ==914== by 0x4E3EBDF: ompi_free_list_grow (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==914== by 0x8C1FA93: mca_btl_sm_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)
> ==914== by 0x8816759: mca_bml_r2_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)
> ==914== by 0x8400274: mca_pml_ob1_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)
> ==914== by 0x4E5B048: ompi_mpi_init (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==914== by 0x4E79F65: PMPI_Init (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==914== by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
> ==914==
> ==917== Conditional jump or move depends on uninitialised value(s)
> ==917== at 0x7BDFD64: mca_mpool_sm_alloc (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)
> ==917== by 0x4E3EBDF: ompi_free_list_grow (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==917== by 0x8C1FA22: mca_btl_sm_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)
> ==917== by 0x8816759: mca_bml_r2_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)
> ==917== by 0x8400274: mca_pml_ob1_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)
> ==917== by 0x4E5B048: ompi_mpi_init (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==917== by 0x4E79F65: PMPI_Init (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==917== by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
> ==917==
> ==917== Conditional jump or move depends on uninitialised value(s)
> ==908== Conditional jump or move depends on uninitialised value(s)
> ==908== at 0x7BDFD64: mca_mpool_sm_alloc (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)
> ==908== by 0x4E3EBDF: ompi_free_list_grow (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==908== by 0x8C1FA22: mca_btl_sm_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)
> ==908== by 0x8816759: mca_bml_r2_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)
> ==908== by 0x8400274: mca_pml_ob1_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)
> ==908== by 0x4E5B048: ompi_mpi_init (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==908== by 0x4E79F65: PMPI_Init (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==908== by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
> ==908==
> ==917== at 0x7BDFD64: mca_mpool_sm_alloc (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)
> ==917== by 0x4E3EBDF: ompi_free_list_grow (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==917== by 0x8C1FA93: mca_btl_sm_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)
> ==917== by 0x8816759: mca_bml_r2_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)
> ==917== by 0x8400274: mca_pml_ob1_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)
> ==917== by 0x4E5B048: ompi_mpi_init (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==911== Conditional jump or move depends on uninitialised value(s)
> ==911== at 0x7BDFD64: mca_mpool_sm_alloc (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)
> ==911== by 0x8C1F0FC: mca_btl_sm_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)
> ==911== by 0x8816759: mca_bml_r2_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)
> ==911== by 0x8400274: mca_pml_ob1_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)
> ==911== by 0x4E5B048: ompi_mpi_init (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==911== by 0x4E79F65: PMPI_Init (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==911== by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
> ==911==
> ==917== by 0x4E79F65: PMPI_Init (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==917== by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
> ==914== Conditional jump or move depends on uninitialised value(s)
> ==914== at 0x7BDFD64: mca_mpool_sm_alloc (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)
> ==914== by 0x8C1F0FC: mca_btl_sm_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)
> ==914== by 0x8816759: mca_bml_r2_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)
> ==914== by 0x8400274: mca_pml_ob1_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)
> ==914== by 0x4E5B048: ompi_mpi_init (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==914== by 0x4E79F65: PMPI_Init (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==914== by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
> ==914==
> ==917==
> ==923== Conditional jump or move depends on uninitialised value(s)
> ==923== at 0x7BDFD64: mca_mpool_sm_alloc (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)
> ==923== by 0x4E3EBDF: ompi_free_list_grow (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==923== by 0x8C1FA22: mca_btl_sm_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)
> ==923== by 0x8816759: mca_bml_r2_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)
> ==923== by 0x8400274: mca_pml_ob1_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)
> ==923== by 0x4E5B048: ompi_mpi_init (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==923== by 0x4E79F65: PMPI_Init (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==923== by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
> ==923==
> ==920== Conditional jump or move depends on uninitialised value(s)
> ==920== at 0x7BDFD64: mca_mpool_sm_alloc (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)
> ==920== by 0x4E3EBDF: ompi_free_list_grow (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==920== by 0x8C1FA22: mca_btl_sm_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)
> ==920== by 0x8816759: mca_bml_r2_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)
> ==920== by 0x8400274: mca_pml_ob1_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)
> ==920== by 0x4E5B048: ompi_mpi_init (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==920== by 0x4E79F65: PMPI_Init (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==920== by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
> ==920==
> ==908== Conditional jump or move depends on uninitialised value(s)
> ==908== at 0x7BDFD64: mca_mpool_sm_alloc (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)
> ==908== by 0x4E3EBDF: ompi_free_list_grow (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==908== by 0x8C1FA93: mca_btl_sm_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)
> ==908== by 0x8816759: mca_bml_r2_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)
> ==908== by 0x8400274: mca_pml_ob1_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)
> ==908== by 0x4E5B048: ompi_mpi_init (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==908== by 0x4E79F65: PMPI_Init (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==908== by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
> ==908==
> ==923== Conditional jump or move depends on uninitialised value(s)
> ==923== at 0x7BDFD64: mca_mpool_sm_alloc (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)
> ==923== by 0x4E3EBDF: ompi_free_list_grow (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==920== Conditional jump or move depends on uninitialised value(s)
> ==920== at 0x7BDFD64: mca_mpool_sm_alloc (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)
> ==920== by 0x4E3EBDF: ompi_free_list_grow (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==920== by 0x8C1FA93: mca_btl_sm_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)
> ==923== by 0x8C1FA93: mca_btl_sm_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)
> ==923== by 0x8816759: mca_bml_r2_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)
> ==923== by 0x8400274: mca_pml_ob1_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)
> ==923== by 0x4E5B048: ompi_mpi_init (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==923== by 0x4E79F65: PMPI_Init (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==923== by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
> ==923==
> ==920== by 0x8816759: mca_bml_r2_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)
> ==920== by 0x8400274: mca_pml_ob1_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)
> ==920== by 0x4E5B048: ompi_mpi_init (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==920== by 0x4E79F65: PMPI_Init (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==920== by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
> ==920==
> ==905== Conditional jump or move depends on uninitialised value(s)
> ==905== at 0x7BDFD64: mca_mpool_sm_alloc (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)
> ==905== by 0x8C1F94D: mca_btl_sm_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)
> ==905== by 0x8816759: mca_bml_r2_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)
> ==905== by 0x8400274: mca_pml_ob1_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)
> ==905== by 0x4E5B048: ompi_mpi_init (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==905== by 0x4E79F65: PMPI_Init (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==905== by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
> ==905==
> ==917== Conditional jump or move depends on uninitialised value(s)
> ==917== at 0x7BDFD64: mca_mpool_sm_alloc (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)
> ==917== by 0x8C1F0FC: mca_btl_sm_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)
> ==917== by 0x8816759: mca_bml_r2_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)
> ==917== by 0x8400274: mca_pml_ob1_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)
> ==917== by 0x4E5B048: ompi_mpi_init (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==917== by 0x4E79F65: PMPI_Init (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==917== by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
> ==917==
> ==908== Conditional jump or move depends on uninitialised value(s)
> ==908== at 0x7BDFD64: mca_mpool_sm_alloc (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)
> ==908== by 0x8C1F0FC: mca_btl_sm_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)
> ==908== by 0x8816759: mca_bml_r2_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)
> ==908== by 0x8400274: mca_pml_ob1_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)
> ==908== by 0x4E5B048: ompi_mpi_init (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==908== by 0x4E79F65: PMPI_Init (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==908== by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
> ==908==
> ==905== Conditional jump or move depends on uninitialised value(s)
> ==905== at 0x7BDFD64: mca_mpool_sm_alloc (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)
> ==905== by 0x4E3EBDF: ompi_free_list_grow (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==905== by 0x8C1FA22: mca_btl_sm_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)
> ==905== by 0x8816759: mca_bml_r2_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)
> ==905== by 0x8400274: mca_pml_ob1_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)
> ==905== by 0x4E5B048: ompi_mpi_init (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==905== by 0x4E79F65: PMPI_Init (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==905== by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
> ==905==
> ==923== Conditional jump or move depends on uninitialised value(s)
> ==923== at 0x7BDFD64: mca_mpool_sm_alloc (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)
> ==923== by 0x8C1F0FC: mca_btl_sm_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)
> ==923== by 0x8816759: mca_bml_r2_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)
> ==923== by 0x8400274: mca_pml_ob1_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)
> ==923== by 0x4E5B048: ompi_mpi_init (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==923== by 0x4E79F65: PMPI_Init (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==923== by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
> ==923==
> ==920== Conditional jump or move depends on uninitialised value(s)
> ==920== at 0x7BDFD64: mca_mpool_sm_alloc (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)
> ==920== by 0x8C1F0FC: mca_btl_sm_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)
> ==920== by 0x8816759: mca_bml_r2_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)
> ==920== by 0x8400274: mca_pml_ob1_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)
> ==920== by 0x4E5B048: ompi_mpi_init (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==920== by 0x4E79F65: PMPI_Init (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==920== by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
> ==920==
> ==905== Conditional jump or move depends on uninitialised value(s)
> ==905== at 0x7BDFD64: mca_mpool_sm_alloc (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)
> ==905== by 0x4E3EBDF: ompi_free_list_grow (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==905== by 0x8C1FA93: mca_btl_sm_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)
> ==905== by 0x8816759: mca_bml_r2_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)
> ==905== by 0x8400274: mca_pml_ob1_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)
> ==905== by 0x4E5B048: ompi_mpi_init (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==905== by 0x4E79F65: PMPI_Init (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==905== by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
> ==905==
> ==917== Conditional jump or move depends on uninitialised value(s)
> ==917== at 0x7BDFD64: mca_mpool_sm_alloc (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)
> ==917== by 0x4E3EBDF: ompi_free_list_grow (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==917== by 0x4E3EC5F: ompi_free_list_resize (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==917== by 0x8C1F630: mca_btl_sm_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)
> ==917== by 0x8816759: mca_bml_r2_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)
> ==917== by 0x8400274: mca_pml_ob1_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)
> ==917== by 0x4E5B048: ompi_mpi_init (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==917== by 0x4E79F65: PMPI_Init (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==917== by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
> ==917==
> ==908== Conditional jump or move depends on uninitialised value(s)
> ==908== at 0x7BDFD64: mca_mpool_sm_alloc (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)
> ==908== by 0x4E3EBDF: ompi_free_list_grow (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==908== by 0x4E3EC5F: ompi_free_list_resize (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==908== by 0x8C1F630: mca_btl_sm_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)
> ==908== by 0x8816759: mca_bml_r2_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)
> ==908== by 0x8400274: mca_pml_ob1_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)
> ==908== by 0x4E5B048: ompi_mpi_init (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==908== by 0x4E79F65: PMPI_Init (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==908== by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
> ==908==
> ==923== Conditional jump or move depends on uninitialised value(s)
> ==923== at 0x7BDFD64: mca_mpool_sm_alloc (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)
> ==923== by 0x4E3EBDF: ompi_free_list_grow (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==923== by 0x4E3EC5F: ompi_free_list_resize (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==923== by 0x8C1F630: mca_btl_sm_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)
> ==923== by 0x8816759: mca_bml_r2_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)
> ==923== by 0x8400274: mca_pml_ob1_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)
> ==923== by 0x4E5B048: ompi_mpi_init (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==923== by 0x4E79F65: PMPI_Init (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==923== by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
> ==923==
> ==920== Conditional jump or move depends on uninitialised value(s)
> ==920== at 0x7BDFD64: mca_mpool_sm_alloc (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)
> ==920== by 0x4E3EBDF: ompi_free_list_grow (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==920== by 0x4E3EC5F: ompi_free_list_resize (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==920== by 0x8C1F630: mca_btl_sm_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)
> ==920== by 0x8816759: mca_bml_r2_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)
> ==920== by 0x8400274: mca_pml_ob1_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)
> ==920== by 0x4E5B048: ompi_mpi_init (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==920== by 0x4E79F65: PMPI_Init (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==920== by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
> ==920==
> ==905== Conditional jump or move depends on uninitialised value(s)
> ==905== at 0x7BDFD64: mca_mpool_sm_alloc (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)
> ==905== by 0x8C1F0FC: mca_btl_sm_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)
> ==905== by 0x8816759: mca_bml_r2_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)
> ==905== by 0x8400274: mca_pml_ob1_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)
> ==905== by 0x4E5B048: ompi_mpi_init (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==905== by 0x4E79F65: PMPI_Init (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==905== by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
> ==905==
> ==905== Conditional jump or move depends on uninitialised value(s)
> ==905== at 0x7BDFD64: mca_mpool_sm_alloc (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)
> ==905== by 0x4E3EBDF: ompi_free_list_grow (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==905== by 0x4E3EC5F: ompi_free_list_resize (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==905== by 0x8C1F630: mca_btl_sm_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)
> ==905== by 0x8816759: mca_bml_r2_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)
> ==905== by 0x8400274: mca_pml_ob1_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)
> ==905== by 0x4E5B048: ompi_mpi_init (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==905== by 0x4E79F65: PMPI_Init (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==905== by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
> ==905==
> ==902== Conditional jump or move depends on uninitialised value(s)
> ==902== at 0x7BDFD64: mca_mpool_sm_alloc (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)
> ==902== by 0x4E3EBDF: ompi_free_list_grow (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==902== by 0x4E3EC5F: ompi_free_list_resize (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==902== by 0x8C1F630: mca_btl_sm_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)
> ==902== by 0x8816759: mca_bml_r2_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)
> ==902== by 0x8400274: mca_pml_ob1_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)
> ==902== by 0x4E5B048: ompi_mpi_init (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==902== by 0x4E79F65: PMPI_Init (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==902== by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
> ==902==
> ==911== Conditional jump or move depends on uninitialised value(s)
> ==911== at 0x7BDFD64: mca_mpool_sm_alloc (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)
> ==911== by 0x4E3EBDF: ompi_free_list_grow (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==911== by 0x4E3EC5F: ompi_free_list_resize (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==911== by 0x8C1F630: mca_btl_sm_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)
> ==911== by 0x8816759: mca_bml_r2_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)
> ==911== by 0x8400274: mca_pml_ob1_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)
> ==911== by 0x4E5B048: ompi_mpi_init (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==911== by 0x4E79F65: PMPI_Init (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==911== by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
> ==911==
> ==914== Conditional jump or move depends on uninitialised value(s)
> ==914== at 0x7BDFD64: mca_mpool_sm_alloc (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)
> ==914== by 0x4E3EBDF: ompi_free_list_grow (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==914== by 0x4E3EC5F: ompi_free_list_resize (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==914== by 0x8C1F630: mca_btl_sm_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)
> ==914== by 0x8816759: mca_bml_r2_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)
> ==914== by 0x8400274: mca_pml_ob1_add_procs (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)
> ==914== by 0x4E5B048: ompi_mpi_init (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==914== by 0x4E79F65: PMPI_Init (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==914== by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
> ==914==
> [octet:00923] *** Process received signal ***
> [octet:00923] Signal: Segmentation fault (11)
> [octet:00923] Signal code: Address not mapped (1)
> [octet:00923] Failing at address: 0x40
> [octet:00923] [ 0] /lib64/libpthread.so.0 [0x3825c0eee0]
> [octet:00923] [ 1]
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so [0x8c21009]
> [octet:00923] [ 2]
> /usr/local/openmpi-1.3.3-valg/lib/libopen-pal.so.0(opal_progress+0x5a)
> [0x5324c9a]
> [octet:00923] [ 3] /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0
> [0x4e59d45]
> [octet:00923] [ 4]
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_coll_tuned.so [0x9ccad2a]
> [octet:00923] [ 5]
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_coll_tuned.so [0x9cd349e]
> [octet:00923] [ 6]
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0(MPI_Barrier+0x72)
> [0x4e6fa32]
> [octet:00923] [ 7] ompi1.3.3-bug(main+0x1ee) [0x409152]
> [octet:00923] [ 8] /lib64/libc.so.6(__libc_start_main+0xfd)
> [0x382501ea2d]
> [octet:00923] [ 9] ompi1.3.3-bug [0x408ea9]
> [octet:00923] *** End of error message ***
> ==923==
> ==923== Process terminating with default action of signal 11 (SIGSEGV)
> ==923== Access not within mapped region at address 0x40
> ==923== at 0x8C21009: mca_btl_sm_component_progress (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)
> ==923== by 0x5324C99: opal_progress (in
> /usr/local/openmpi-1.3.3-valg/lib/libopen-pal.so.0.0.0)
> ==923== by 0x4E59D44: ompi_request_default_wait_all (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==923== by 0x9CCAD29: ompi_coll_tuned_sendrecv_actual (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_coll_tuned.so)
> ==923== by 0x9CD349D:
> ompi_coll_tuned_barrier_intra_recursivedoubling (in
> /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_coll_tuned.so)
> ==923== by 0x4E6FA31: PMPI_Barrier (in
> /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
> ==923== by 0x409151: main (in /home/research/cary/ompi1.3.3-bug)
> ==923== If you believe this happened as a result of a stack
> ==923== overflow in your program's main thread (unlikely but
> ==923== possible), you can try to increase the size of the
> ==923== main thread stack using the --main-stacksize= flag.
> ==923== The main thread stack size used in this run was 10485760.
> ==923==
> ==923== HEAP SUMMARY:
> ==923== in use at exit: 2,062,803 bytes in 2,652 blocks
> ==923== total heap usage: 10,244 allocs, 7,592 frees, 15,824,029
> bytes allocated
> ==923==
> ==923== LEAK SUMMARY:
> ==923== definitely lost: 1,700 bytes in 29 blocks
> ==923== indirectly lost: 2,400 bytes in 1 blocks
> ==923== possibly lost: 2,384 bytes in 22 blocks
> ==923== still reachable: 2,056,319 bytes in 2,600 blocks
> ==923== suppressed: 0 bytes in 0 blocks
> ==923== Rerun with --leak-check=full to see details of leaked memory
> ==923==
> ==923== For counts of detected and suppressed errors, rerun with: -v
> ==923== Use --track-origins=yes to see where uninitialised values come
> from
> ==923== ERROR SUMMARY: 5 errors from 5 contexts (suppressed: 7 from 7)
> --------------------------------------------------------------------------
>
> mpirun noticed that process rank 7 with PID 923 on node
> octet.carys.home exited on signal 11 (Segmentation fault).
> --------------------------------------------------------------------------
>
>
>

-- 
Tech-X Corp., 5621 Arapahoe Ave, Suite A, Boulder CO 80303
cary_at_[hidden], p 303-448-0727, f 303-448-7756, NEW CELL 303-881-8572