Open MPI logo

Open MPI Development Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Development mailing list

Subject: Re: [OMPI devel] Trunk is broken
From: Ralph Castain (rhc_at_[hidden])
Date: 2014-02-08 16:57:41


Temporary workaround: -mca btl ^vader

On Feb 8, 2014, at 10:11 AM, Ralph Castain <rhc_at_[hidden]> wrote:

> Sorry to say, some recent commit has broken the trunk:
>
> rhc_at_bend002 examples]$ mpirun -n 3 ./hello_c
> [bend001:22289] *** Process received signal ***
> [bend001:22289] Signal: Segmentation fault (11)
> [bend001:22289] Signal code: Invalid permissions (2)
> [bend001:22289] Failing at address: 0x7f354daaa000
> [bend001:22290] *** Process received signal ***
> [bend001:22290] Signal: Segmentation fault (11)
> [bend001:22290] Signal code: Invalid permissions (2)
> [bend001:22290] Failing at address: 0x7fa819d81000
> [bend001:22289] [ 0] /lib64/libpthread.so.0[0x38e320f710]
> [bend001:22289] [ 1] /lib64/libc.so.6[0x38e26845ad]
> [bend001:22289] [ 2] /home/common/openmpi/build/svn-trunk/lib/openmpi/mca_btl_vader.so(+0x3b0b)[0x7f3549924b0b]
> [bend001:22289] [ 3] /home/common/openmpi/build/svn-trunk/lib/libmpi.so.0(mca_btl_base_select+0x1cc)[0x7f354db62a21]
> [bend001:22289] [ 4] /home/common/openmpi/build/svn-trunk/lib/openmpi/mca_bml_r2.so(mca_bml_r2_component_init+0x27)[0x7f354a1cfc2c]
> [bend001:22289] [ 5] [bend001:22290] [ 0] /lib64/libpthread.so.0[0x38e320f710]
> [bend001:22290] [ 1] /lib64/libc.so.6[0x38e26845ad]
> [bend001:22290] [ 2] /home/common/openmpi/build/svn-trunk/lib/openmpi/mca_btl_vader.so(+0x3b0b)[0x7fa815bfbb0b]
> [bend001:22290] [ 3] /home/common/openmpi/build/svn-trunk/lib/libmpi.so.0(mca_bml_base_init+0xe2)[0x7f354db6189e]
> [bend001:22289] [ 6] /home/common/openmpi/build/svn-trunk/lib/openmpi/mca_pml_ob1.so(+0x7cc3)[0x7f35492c3cc3]
> [bend001:22289] [ 7] /home/common/openmpi/build/svn-trunk/lib/libmpi.so.0(mca_pml_base_select+0x29c)[0x7f354db88261]
> [bend001:22289] [ 8] /home/common/openmpi/build/svn-trunk/lib/libmpi.so.0(ompi_mpi_init+0x685)[0x7f354dafbc7b]
> [bend001:22289] [ 9] /home/common/openmpi/build/svn-trunk/lib/libmpi.so.0(mca_btl_base_select+0x1cc)[0x7fa819e39a21]
> [bend001:22290] [ 4] /home/common/openmpi/build/svn-trunk/lib/openmpi/mca_bml_r2.so(mca_bml_r2_component_init+0x27)[0x7fa8164a6c2c]
> [bend001:22290] [ 5] /home/common/openmpi/build/svn-trunk/lib/libmpi.so.0(mca_bml_base_init+0xe2)[0x7fa819e3889e]
> [bend001:22290] [ 6] /home/common/openmpi/build/svn-trunk/lib/openmpi/mca_pml_ob1.so(+0x7cc3)[0x7fa81559acc3]
> [bend001:22290] [ 7] /home/common/openmpi/build/svn-trunk/lib/libmpi.so.0(mca_pml_base_select+0x29c)[0x7fa819e5f261]
> [bend001:22290] [ 8] /home/common/openmpi/build/svn-trunk/lib/libmpi.so.0(MPI_Init+0x185)[0x7f354db2f156]
> [bend001:22289] [10] ./hello_c[0x400806]
> [bend001:22289] [11] /lib64/libc.so.6(__libc_start_main+0xfd)[0x38e261ed1d]
> [bend001:22289] [12] ./hello_c[0x400719]
> [bend001:22289] *** End of error message ***
> /home/common/openmpi/build/svn-trunk/lib/libmpi.so.0(ompi_mpi_init+0x685)[0x7fa819dd2c7b]
> [bend001:22290] [ 9] /home/common/openmpi/build/svn-trunk/lib/libmpi.so.0(MPI_Init+0x185)[0x7fa819e06156]
> [bend001:22290] [10] ./hello_c[0x400806]
> [bend001:22290] [11] /lib64/libc.so.6(__libc_start_main+0xfd)[0x38e261ed1d]
> [bend001:22290] [12] ./hello_c[0x400719]
> [bend001:22290] *** End of error message ***
> [bend001:22291] *** Process received signal ***
> [bend001:22291] Signal: Segmentation fault (11)
> [bend001:22291] Signal code: Invalid permissions (2)
> [bend001:22291] Failing at address: 0x7f498fc96000
> [bend001:22291] [ 0] /lib64/libpthread.so.0[0x38e320f710]
> [bend001:22291] [ 1] /lib64/libc.so.6[0x38e26845ad]
> [bend001:22291] [ 2] /home/common/openmpi/build/svn-trunk/lib/openmpi/mca_btl_vader.so(+0x3b0b)[0x7f498795db0b]
> [bend001:22291] [ 3] /home/common/openmpi/build/svn-trunk/lib/libmpi.so.0(mca_btl_base_select+0x1cc)[0x7f498fd4ea21]
> [bend001:22291] [ 4] /home/common/openmpi/build/svn-trunk/lib/openmpi/mca_bml_r2.so(mca_bml_r2_component_init+0x27)[0x7f498c3bbc2c]
> [bend001:22291] [ 5] /home/common/openmpi/build/svn-trunk/lib/libmpi.so.0(mca_bml_base_init+0xe2)[0x7f498fd4d89e]
> [bend001:22291] [ 6] /home/common/openmpi/build/svn-trunk/lib/openmpi/mca_pml_ob1.so(+0x7cc3)[0x7f49872fccc3]
> [bend001:22291] [ 7] /home/common/openmpi/build/svn-trunk/lib/libmpi.so.0(mca_pml_base_select+0x29c)[0x7f498fd74261]
> [bend001:22291] [ 8] /home/common/openmpi/build/svn-trunk/lib/libmpi.so.0(ompi_mpi_init+0x685)[0x7f498fce7c7b]
> [bend001:22291] [ 9] /home/common/openmpi/build/svn-trunk/lib/libmpi.so.0(MPI_Init+0x185)[0x7f498fd1b156]
> [bend001:22291] [10] ./hello_c[0x400806]
> [bend001:22291] [11] /lib64/libc.so.6(__libc_start_main+0xfd)[0x38e261ed1d]
> [bend001:22291] [12] ./hello_c[0x400719]
> [bend001:22291] *** End of error message ***
> --------------------------------------------------------------------------
> mpirun noticed that process rank 0 with PID 22289 on node bend001 exited on signal 11 (Segmentation fault).
> --------------------------------------------------------------------------
> 3 total processes killed (some possibly by mpirun during cleanup)
> [rhc_at_bend002 examples]$
>
> Nathan: can you please take a look?
>
> Ralph
>