Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: Re: [OMPI users] Segmentation fault in mca_btl_tcp
From: Ake Sandgren (ake.sandgren_at_[hidden])
Date: 2010-04-15 03:18:32


On Thu, 2010-04-15 at 15:57 +0900, Werner Van Geit wrote:
> Hi,
>
> We are using openmpi 1.4.1 on our cluster computer (in conjunction with Torque). One of our users has a problem with his jobs generating a segmentation fault on one of the slaves, this is the backtrace:
>
> [cstone-00613:28461] *** Process received signal ***
> [cstone-00613:28461] Signal: Segmentation fault (11)
> [cstone-00613:28461] Signal code: (128)
> [cstone-00613:28461] Failing at address: (nil)
> [cstone-00613:28462] *** Process received signal ***
> [cstone-00613:28462] Signal: Segmentation fault (11)
> [cstone-00613:28462] Signal code: Address not mapped (1)
> [cstone-00613:28462] Failing at address: (nil)
> [cstone-00613:28461] [ 0] /lib64/libc.so.6 [0x2ba1933dce20]
> [cstone-00613:28461] [ 1] /opt/openmpi-1.3/lib/openmpi/mca_btl_tcp.so [0x2ba19530ec7a]
> [cstone-00613:28461] [ 2] /opt/openmpi-1.3/lib/openmpi/mca_btl_tcp.so [0x2ba19530d860]
> [cstone-00613:28461] [ 3] /opt/openmpi/lib/libopen-pal.so.0 [0x2ba1938eb16b]
> [cstone-00613:28461] [ 4] /opt/openmpi/lib/libopen-pal.so.0(opal_progress+0x9e) [0x2ba1938e072e]
> [cstone-00613:28461] [ 5] /opt/openmpi/lib/libmpi.so.0 [0x2ba193621b38]
> [cstone-00613:28461] [ 6] /opt/openmpi/lib/libmpi.so.0(PMPI_Wait+0x5b) [0x2ba19364c63b]
> [cstone-00613:28461] [ 7] /opt/openmpi/lib/libmpi_f77.so.0(mpi_wait_+0x3a) [0x2ba192e98b8a]
> [cstone-00613:28461] [ 8] ./roms [0x44976c]
> [cstone-00613:28461] [ 9] ./roms [0x449d96]
> [cstone-00613:28461] [10] ./roms [0x422708]
> [cstone-00613:28461] [11] ./roms [0x402908]
> [cstone-00613:28461] [12] ./roms [0x402467]
> [cstone-00613:28461] [13] ./roms [0x46d20e]
> [cstone-00613:28461] [14] /lib64/libc.so.6(__libc_start_main+0xf4) [0x2ba1933ca164]
> [cstone-00613:28461] [15] ./roms [0x401dd9]
> [cstone-00613:28461] *** End of error message ***
> [cstone-00613:28462] [ 0] /lib64/libc.so.6 [0x2b5d57db6e20]
> [cstone-00613:28462] *** End of error message ***
>
> The other slaves crash with:
> [cstone-00612][[21785,1],35][btl_tcp_frag.c:216:mca_btl_tcp_frag_recv] mca_btl_tcp_frag_recv: readv failed: Connection reset by peer (104)
>
> Since this problem seems to be happening in the network part of MPI my guess is that there is, or something wrong with the network, or a bug in OpenMPI.
> This same problem also appeared at the time that we were using openmpi 1.3
>
> How could this problem be solved ?

We sometimes see mysterious crashes like this one. At least some of them
are caused by port scanners, i.e. unexpected non-mpi related packets
coming in on the sockets will sometimes cause havoc.

We've been getting http traffic in the jobs stdout/err sometimes. That
really makes the users confused :-)

And yes, we are going to block this but we haven't had time...

-- 
Ake Sandgren, HPC2N, Umea University, S-90187 Umea, Sweden
Internet: ake_at_[hidden]   Phone: +46 90 7866134 Fax: +46 90 7866126
Mobile: +46 70 7716134 WWW: http://www.hpc2n.umu.se