Temporary workaround:  -mca btl ^vader

On Feb 8, 2014, at 10:11 AM, Ralph Castain <rhc@open-mpi.org> wrote:

Sorry to say, some recent commit has broken the trunk:

rhc@bend002 examples]$ mpirun -n 3 ./hello_c
[bend001:22289] *** Process received signal ***
[bend001:22289] Signal: Segmentation fault (11)
[bend001:22289] Signal code: Invalid permissions (2)
[bend001:22289] Failing at address: 0x7f354daaa000
[bend001:22290] *** Process received signal ***
[bend001:22290] Signal: Segmentation fault (11)
[bend001:22290] Signal code: Invalid permissions (2)
[bend001:22290] Failing at address: 0x7fa819d81000
[bend001:22289] [ 0] /lib64/libpthread.so.0[0x38e320f710]
[bend001:22289] [ 1] /lib64/libc.so.6[0x38e26845ad]
[bend001:22289] [ 2] /home/common/openmpi/build/svn-trunk/lib/openmpi/mca_btl_vader.so(+0x3b0b)[0x7f3549924b0b]
[bend001:22289] [ 3] /home/common/openmpi/build/svn-trunk/lib/libmpi.so.0(mca_btl_base_select+0x1cc)[0x7f354db62a21]
[bend001:22289] [ 4] /home/common/openmpi/build/svn-trunk/lib/openmpi/mca_bml_r2.so(mca_bml_r2_component_init+0x27)[0x7f354a1cfc2c]
[bend001:22289] [ 5] [bend001:22290] [ 0] /lib64/libpthread.so.0[0x38e320f710]
[bend001:22290] [ 1] /lib64/libc.so.6[0x38e26845ad]
[bend001:22290] [ 2] /home/common/openmpi/build/svn-trunk/lib/openmpi/mca_btl_vader.so(+0x3b0b)[0x7fa815bfbb0b]
[bend001:22290] [ 3] /home/common/openmpi/build/svn-trunk/lib/libmpi.so.0(mca_bml_base_init+0xe2)[0x7f354db6189e]
[bend001:22289] [ 6] /home/common/openmpi/build/svn-trunk/lib/openmpi/mca_pml_ob1.so(+0x7cc3)[0x7f35492c3cc3]
[bend001:22289] [ 7] /home/common/openmpi/build/svn-trunk/lib/libmpi.so.0(mca_pml_base_select+0x29c)[0x7f354db88261]
[bend001:22289] [ 8] /home/common/openmpi/build/svn-trunk/lib/libmpi.so.0(ompi_mpi_init+0x685)[0x7f354dafbc7b]
[bend001:22289] [ 9] /home/common/openmpi/build/svn-trunk/lib/libmpi.so.0(mca_btl_base_select+0x1cc)[0x7fa819e39a21]
[bend001:22290] [ 4] /home/common/openmpi/build/svn-trunk/lib/openmpi/mca_bml_r2.so(mca_bml_r2_component_init+0x27)[0x7fa8164a6c2c]
[bend001:22290] [ 5] /home/common/openmpi/build/svn-trunk/lib/libmpi.so.0(mca_bml_base_init+0xe2)[0x7fa819e3889e]
[bend001:22290] [ 6] /home/common/openmpi/build/svn-trunk/lib/openmpi/mca_pml_ob1.so(+0x7cc3)[0x7fa81559acc3]
[bend001:22290] [ 7] /home/common/openmpi/build/svn-trunk/lib/libmpi.so.0(mca_pml_base_select+0x29c)[0x7fa819e5f261]
[bend001:22290] [ 8] /home/common/openmpi/build/svn-trunk/lib/libmpi.so.0(MPI_Init+0x185)[0x7f354db2f156]
[bend001:22289] [10] ./hello_c[0x400806]
[bend001:22289] [11] /lib64/libc.so.6(__libc_start_main+0xfd)[0x38e261ed1d]
[bend001:22289] [12] ./hello_c[0x400719]
[bend001:22289] *** End of error message ***
/home/common/openmpi/build/svn-trunk/lib/libmpi.so.0(ompi_mpi_init+0x685)[0x7fa819dd2c7b]
[bend001:22290] [ 9] /home/common/openmpi/build/svn-trunk/lib/libmpi.so.0(MPI_Init+0x185)[0x7fa819e06156]
[bend001:22290] [10] ./hello_c[0x400806]
[bend001:22290] [11] /lib64/libc.so.6(__libc_start_main+0xfd)[0x38e261ed1d]
[bend001:22290] [12] ./hello_c[0x400719]
[bend001:22290] *** End of error message ***
[bend001:22291] *** Process received signal ***
[bend001:22291] Signal: Segmentation fault (11)
[bend001:22291] Signal code: Invalid permissions (2)
[bend001:22291] Failing at address: 0x7f498fc96000
[bend001:22291] [ 0] /lib64/libpthread.so.0[0x38e320f710]
[bend001:22291] [ 1] /lib64/libc.so.6[0x38e26845ad]
[bend001:22291] [ 2] /home/common/openmpi/build/svn-trunk/lib/openmpi/mca_btl_vader.so(+0x3b0b)[0x7f498795db0b]
[bend001:22291] [ 3] /home/common/openmpi/build/svn-trunk/lib/libmpi.so.0(mca_btl_base_select+0x1cc)[0x7f498fd4ea21]
[bend001:22291] [ 4] /home/common/openmpi/build/svn-trunk/lib/openmpi/mca_bml_r2.so(mca_bml_r2_component_init+0x27)[0x7f498c3bbc2c]
[bend001:22291] [ 5] /home/common/openmpi/build/svn-trunk/lib/libmpi.so.0(mca_bml_base_init+0xe2)[0x7f498fd4d89e]
[bend001:22291] [ 6] /home/common/openmpi/build/svn-trunk/lib/openmpi/mca_pml_ob1.so(+0x7cc3)[0x7f49872fccc3]
[bend001:22291] [ 7] /home/common/openmpi/build/svn-trunk/lib/libmpi.so.0(mca_pml_base_select+0x29c)[0x7f498fd74261]
[bend001:22291] [ 8] /home/common/openmpi/build/svn-trunk/lib/libmpi.so.0(ompi_mpi_init+0x685)[0x7f498fce7c7b]
[bend001:22291] [ 9] /home/common/openmpi/build/svn-trunk/lib/libmpi.so.0(MPI_Init+0x185)[0x7f498fd1b156]
[bend001:22291] [10] ./hello_c[0x400806]
[bend001:22291] [11] /lib64/libc.so.6(__libc_start_main+0xfd)[0x38e261ed1d]
[bend001:22291] [12] ./hello_c[0x400719]
[bend001:22291] *** End of error message ***
--------------------------------------------------------------------------
mpirun noticed that process rank 0 with PID 22289 on node bend001 exited on signal 11 (Segmentation fault).
--------------------------------------------------------------------------
3 total processes killed (some possibly by mpirun during cleanup)
[rhc@bend002 examples]$ 

Nathan: can you please take a look?

Ralph