I have been testing some code against openmpi lately that always causes
it to crash during certain mpi function calls. The code does not seem
to be the problem, as it runs just fine against mpich. I have tested
it against openmpi 1.2.5, 1.2.6, and 1.2.7 and they all exhibit the
same problem. Also, the problem only occurs in openmpi when running
more than 16 processes. I have posted this stack trace to the list
before, but I am submitting it now as a potential bug report. I need
some help debugging it and finding out exactly what is going on in
openmpi when the segfault occurs. Are there any suggestions on how
best to do this? Is there an easy way to attach gdb to one of the
processes or something?? I have already compiled openmpi with
debugging, memory profiling, etc. How can I best take advantage of
these features?
Here is the segfault:
[m4b-1-8:11481] *** Process received signal ***
[m4b-1-8:11481] Signal: Segmentation fault (11)
[m4b-1-8:11481] Signal code: Address not mapped (1)
[m4b-1-8:11481] Failing at address: 0x2b91c69eed
[m4b-1-8:11483] [ 0] /lib64/libpthread.so.0 [0x33e8c0de70]
[m4b-1-8:11483] [ 1] /fslhome/dhansen7/openmpi/lib/libmpi.so.0 [0x2aaaaabea7c0]
[m4b-1-8:11483] [ 2] /fslhome/dhansen7/openmpi/lib/libmpi.so.0 [0x2aaaaabea675]
[m4b-1-8:11483] [ 3] /fslhome/dhansen7/openmpi/lib/libmpi.so.0(mca_pml_ob1_send+0x2da) [0x2aaaaabeaf55]
[m4b-1-8:11483] [ 4] /fslhome/dhansen7/openmpi/lib/libmpi.so.0(MPI_Send+0x28e) [0x2aaaaab52c5a]
[m4b-1-8:11483] [ 5] /fslhome/dhansen7/compute/for_DanielHansen/replica_mpi_marylou2/Openmpi_md_twham(twham_init+0x708) [0x42a8a8]
[m4b-1-8:11483] [ 6] /fslhome/dhansen7/compute/for_DanielHansen/replica_mpi_marylou2/Openmpi_md_twham(repexch+0x73c) [0x425d5c]
[m4b-1-8:11483] [ 7] /fslhome/dhansen7/compute/for_DanielHansen/replica_mpi_marylou2/Openmpi_md_twham(main+0x855) [0x4133a5]
[m4b-1-8:11483] [ 8] /lib64/libc.so.6(__libc_start_main+0xf4) [0x33e841d8a4]
[m4b-1-8:11483] [ 9] /fslhome/dhansen7/compute/for_DanielHansen/replica_mpi_marylou2/Openmpi_md_twham [0x4040b9]
[m4b-1-8:11483] *** End of error message ***
There is also an mpi error that just precedes the segfault:
[m4b-1-7i][0,1,0][btl_tcp_component.c:623:mca_btl_tcp_component_recv_handler] errno=11
[m4b-1-7i][0,1,0][btl_tcp_component.c:623:mca_btl_tcp_component_recv_handler] errno=11
[m4b-1-7i][0,1,0][btl_tcp_component.c:623:mca_btl_tcp_component_recv_handler] errno=11
[m4b-1-7i][0,1,0][btl_tcp_component.c:623:mca_btl_tcp_component_recv_handler] errno=11
Thanks,
Daniel Hansen
Systems Administrator
BYU Fulton Supercomputing Lab