Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: Re: [OMPI users] segmentation fault: Address not mapped
From: Jeff Squyres (jsquyres_at_[hidden])
Date: 2009-11-02 21:19:03


Many thanks for all this information. Unfortunately, it's not enough
to know what's going on. :-(

Do you know for sure that the application is correct? E.g., is it
possible that a bad buffer is being passed to MPI_Isend? I note that
it is fairly odd to fail in MPI_Isend itself because that function is
actually pretty short -- it mainly checks parameters and then calls a
back-end Open MPI function to actually do the send.

Do you get corefiles with the killed processes, and can you analyze
where the application failed? If so, can you verify that all state in
the application appears to be correct? It might be helpful to analyze
exactly where the application failed (e.g., compile at least ompi/mpi/
c/isend.c with the -g flag so that you can get some debugging
information about exactly where in MPI_Isend it failed -- like I said,
it's a short function that mainly checks parameters). You might want
to have your application double check all the parameters that are
passed to MPI_Isend, too.

On Oct 26, 2009, at 9:43 AM, Iris Pernille Lohmann wrote:

> Dear list members
>
> I am using openmpi 1.3.3 with OFED on a HP cluster with redhatLinux.
>
> Occasionally (not always) I get a crash with the following message:
>
> [hydra11:09312] *** Process received signal ***
> [hydra11:09312] Signal: Segmentation fault (11)
> [hydra11:09312] Signal code: Address not mapped (1)
> [hydra11:09312] Failing at address: 0xffffffffab5f30a8
> [hydra11:09312] [ 0] /lib64/libpthread.so.0 [0x3c1400e4c0]
> [hydra11:09312] [ 1] /home/ipl/openmpi-1.3.3/platforms/hp/lib/
> libmpi.so.0(MPI_Isend+0x93) [0x2af1be45a3e3]
> [hydra11:09312] [ 2] ./flow(MP_SendReal+0x60) [0x6bc993]
> [hydra11:09312] [ 3] ./flow(SendRealsAlongFaceWithOffset_3D+0x4ab)
> [0x68ba19]
> [hydra11:09312] [ 4] ./flow(MP_SendVertexArrayBlock+0x23d) [0x6891e1]
> [hydra11:09312] [ 5] ./flow(MB_CommAllVertex+0x65) [0x6848ba]
> [hydra11:09312] [ 6] ./flow(MB_SetupVertexArray+0xd5) [0x68c837]
> [hydra11:09312] [ 7] ./flow(MB_SetupGrid+0xa8) [0x68be51]
> [hydra11:09312] [ 8] ./flow(SetGrid+0x58) [0x446224]
> [hydra11:09312] [ 9] ./flow(main+0x148) [0x43b728]
> [hydra11:09312] [10] /lib64/libc.so.6(__libc_start_main+0xf4)
> [0x3c1341d974]
> [hydra11:09312] [11] ./flow(__gxx_personality_v0+0xd9) [0x429b19]
> [hydra11:09312] *** End of error message ***
> --------------------------------------------------------------------------
> mpirun noticed that process rank 6 with PID 9312 on node hydra11
> exited on signal 11 (Segmentation fault).
> --------------------------------------------------------------------------
>
> The crash does not appear always – sometimes the application runs
> fine. However, it seems that the crash especially occurs when I run
> on more than 1 node.
>
> I have consulted the archive of open-mpi and have found many error
> messages of the same kind, but none from the 1.3.3 version, and none
> of direct relevance.
>
> I would really appreciate comments on this. Below is the information
> required according to the openmpi web,
>
> Config.log: attached (config.zip)
> Open mpi was configured with prefix and with the path to openib, and
> with the following compiler flags
> setenv CC gcc
> setenv CFLAGS '-O'
> setenv CXX g++
> setenv CXXFLAGS '-O'
> setenv F77 'gfortran'
> setenv FFLAGS '-O'
>
> ompi_info –all:
> attached
>
> The application (named flow) was launched on hydra11 by
> nohup mpirun –H hydra11,hydra12 –np 8 ./flow caseC.in &
>
> the PATH and LD_LIBRARY_PATH, hydra11 and hydra12:
> PATH=/home/ipl/openmpi-1.3.3/platforms/hp/bin
> LD_LIBRARY_PATH= /home/ipl/openmpi-1.3.3/platforms/hp/lib
>
> OpenFabrics version: 1.4
>
> Linux:
> X86_64-redhat-linux/3.4.6
>
> ibv_devinfo, hydra11: attached
> ibv_devinfo, hydra12: attached
>
> ifconfig, hydra11: attached
> ifconfig, hydra12: attached
>
> ulimit –l (hydra11): 6000000
> ulimit –l (hydra12): unlimited
>
> Furthermore, I can say that I have not specified any MCA parameters.
>
> The application which I am running (named flow) is linked from
> fortran, c and c++ libraries with the following:
> /home/ipl/openmpi-1.3.3/platforms/hp/bin/mpicc -DMP -
> DNS3_ARCH_LINUX -DLAPACK -I/home/ipl/ns3/engine/include_forLinux -I/
> home/ipl/openmpi-1.3.3/platforms/hp/include -c -o user_small_3D.o
> user_small_3D.c
> rm -f flow
> /home/ipl/openmpi-1.3.3/platforms/hp/bin/mpicxx -o flow
> user_small_3D.o -L/home/ipl/ns3/engine/lib_forLinux -lns3main -
> lns3pars -lns3util -lns3vofl -lns3turb -lns3solv -lns3mesh -lns3diff
> -lns3grid -lns3line -lns3data -lns3base -lfitpack -lillusolve -
> lfftpack_small -lfenton -lns3air -lns3dens -lns3poro -lns3sedi -
> llapack_small -lblas_small -lm -lgfortran /home/ipl/ns3/engine/
> lib_Tecplot_forLinux/tecio64.a
>
> Please let me know if you need more info!
>
> Thanks in advance,
> Iris Lohmann
>
>
>
>
> Iris Pernille Lohmann
> MSc, PhD
> Ports & Offshore Technology (POT)
>
> <image001.gif>
>
> DHI
> Agern Allé 5
> DK-2970 Hørsholm
> Denmark
>
> Tel:
>
> +45 4516 9200
> Direct:
>
> 45169427
>
> ipl_at_[hidden]
> www.dhigroup.com
>
> WATER • ENVIRONMENT • HEALTH
>
>
> *****************************************************************************
> ** **
> ** WARNING: This email contains an attachment of a very suspicious
> type. **
> ** You are urged NOT to open this attachment unless you are
> absolutely **
> ** sure it is legitimate. Opening this attachment may cause
> irreparable **
> ** damage to your computer and your files. If you have any
> questions **
> ** about the validity of this message, PLEASE SEEK HELP BEFORE
> OPENING IT. **
> ** **
> ** This warning was added by the IU Computer Science Dept. mail
> scanner. **
> *****************************************************************************
>
> <
> config
> .zip
> >
> <
> ompi_info_all
> .zip
> >
> <
> ibv_devinfo_hydra11
> .out
> >
> <
> ibv_devinfo_hydra12
> .out
> >
> <
> ifconfig_hydra11
> .out
> ><ifconfig_hydra12.out>_______________________________________________
> users mailing list
> users_at_[hidden]
> http://www.open-mpi.org/mailman/listinfo.cgi/users

-- 
Jeff Squyres
jsquyres_at_[hidden]