Dear Open MPI User's and Developers,
I encountered a problem with Open MPI when porting an application, which
successfully ran with LAM MPI and MPICH.
The program produces a segmentation fault (see  for the stack trace) when
doing the MPI_Send with the following arguments:
MPI_Send((void *)0, 1, datatype, rank, tag, comm_);
The first argument seems to be wrong at first sight, but is correct because
the argument "datatype" is an MPI_Datatype,
which describes the memory layout of the object to be sent and is
zero-based. The other arguments are as expected: one such object is sent to
rank "rank" with tag "tag" with the help of the communicator "comm_". The
MPI_Datatype is constructed programmatically from the objects member
definitions using MPI_Type_struct. The MPI types involved are solely
MPI_DOUBLE and MPI_UNSIGNED_INT.
I can reproduce the problem with the stable 1.2 release as well as the
1.2.1a snapshot of Open MPI.
My OS is Linux with Kernel 2.6.18 (Debian Etch) running on standard Dual
Xeon Hardware with GigE.
I tried to reduce the amount of data sent by excluding some of the object's
members from the transmission. There does not seem to be a certain member or
type which causes the problem. There seems to be a limit of
members/data/size which determines the success of the call. The "datatype"
structure describes the type and location of approx. 2'000'000 numbers. The
data itself is approx. 16MB (2M * 8 bytes/number assuming doubles), which I
expect not to cause any problem to a MPI implementation.
Thank you for hints, ideas or suggestions where the problem could be.
[head:09133] *** Process received signal ***
[head:09133] Signal: Segmentation fault (11)
[head:09133] Signal code: Address not mapped (1)
[head:09133] Failing at address: 0xb0127475
[head:09133] [ 0] [0xb7f0f440]
[head:09133] [ 1] /usr/lib/libmpi.so.0(ompi_convertor_pack+0x90)
[head:09133] [ 2]
[head:09133] [ 3]
[head:09133] [ 4] /usr/lib/openmpi/mca_pml_ob1.so [0xb5728238]
[head:09133] [ 5] /usr/lib/openmpi/mca_btl_tcp.so [0xb56ddc65]
[head:09133] [ 6] /usr/lib/libopen-pal.so.0(opal_event_base_loop+0x462)
[head:09133] [ 7] /usr/lib/libopen-pal.so.0(opal_event_loop+0x29)
[head:09133] [ 8] /usr/lib/libopen-pal.so.0(opal_progress+0xc0) [0xb65b7260]
[head:09133] [ 9] /usr/lib/openmpi/mca_pml_ob1.so(mca_pml_ob1_send+0x3e5)
[head:09133]  /usr/lib/libmpi.so.0(MPI_Send+0x12f) [0xb66abf0f]
[head:09133]  /opt/plato/release_1.0/bin/engine(main+0x1d23) [0x81617f7]
[head:09133]  /lib/tls/i686/cmov/libc.so.6(__libc_start_main+0xc8)
[head:09133] *** End of error message ***
mpirun noticed that job rank 0 with PID 9133 on node head exited on signal
11 (Segmentation fault).
2 additional processes aborted (not shown)