Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: [OMPI users] Code Seg Faults in Devel Series
From: Doug Roberts (roberpj_at_[hidden])
Date: 2008-06-27 04:10:10


Hi, I am trying to use the latest release of v1.3 to test with BLCR
however i just noticed that sometime after 1.3a1r18423 the standard
mpich sample code (cpi.c) stopped working on our rel4 based myrinet
gm clusters which raises some concern.

Please find attached: gm_board_info.out, ompi_info--all.out,
ompi_info--param-btl-gm.out and config-1.4a1r18743.log bundled
in mpi-output.tar.gz for your analysis.

Below shows the sample code runs with 1.3a1r18423, but crashes with
1.3a1r18740 and further crashes with all snapshots greater than
1.3a1r18423 i have tested. Both gnu 4.2.1 and 4.2.2 were used to
build openmpi and yield the same results. I have not tried 4.2.4
yet, but could next if this looks like a compiler related issue ?
Note also the code runs with both 1.2.6 and 1.2.7rc1 but fails in
all 1.4 releases of the devel trunk.

I also tried the code on a debian myrinet/mx cluster and in contrast
it runs without errors for all 1.3 and 1.4 releases, suggesting the
problem is related to some compatibility issue between open mpi and
our rel4 clusters software stack. Or possibly some parameters simply
need to be applied to the mpirun command that i am not aware of ?

1.3a1r18423
-----------
# /opt/testing/openmpi/1.3a1r18423/bin/mpicc cpi.c
# /opt/testing/openmpi/1.3a1r18423/bin/mpirun -np 12 --host \
ic-bru25,ic-bru27,ic-bru29 a.out
Process 4 of 12 is on bru27
Process 7 of 12 is on bru27
Process 10 of 12 is on bru27
Process 5 of 12 is on bru29
Process 8 of 12 is on bru29
Process 11 of 12 is on bru29
Process 0 of 12 is on bru25
Process 3 of 12 is on bru25
Process 9 of 12 is on bru25
Process 2 of 12 is on bru29
Process 1 of 12 is on bru27
Process 6 of 12 is on bru25
pi is approximately 3.1415926544231252, Error is 0.0000000008333321
wall clock time = 0.013029

1.3a1r18740
-----------
# /opt/testing/openmpi/1.3a1r18740/bin/mpicc cpi.c
# /opt/testing/openmpi/1.3a1r18740/bin/mpirun -np 12 --host \
ic-bru25,ic-bru27,ic-bru29 a.out
Process 5 of 12 is on bru29
Process 8 of 12 is on bru29
Process 11 of 12 is on bru29
Process 1 of 12 is on bru27
Process 4 of 12 is on bru27
Process 7 of 12 is on bru27
Process 10 of 12 is on bru27
Process 0 of 12 is on bru25
[bru25:11224] *** Process received signal ***
[bru25:11224] Signal: Segmentation fault (11)
[bru25:11224] Signal code: Address not mapped (1)
[bru25:11224] Failing at address: 0x9
Process 3 of 12 is on bru25
Process 6 of 12 is on bru25
Process 9 of 12 is on bru25
[bru25:11224] [ 0] /lib64/tls/libpthread.so.0 [0x38c090c420]
[bru25:11224] [ 1]
/opt/testing/openmpi/1.3a1r18740/lib/openmpi/mca_btl_gm.so [0x2a9707ffb9]
[bru25:11224] [ 2]
/opt/testing/openmpi/1.3a1r18740/lib/openmpi/mca_pml_ob1.so [0x2a96d71c1d]
[bru25:11224] [ 3]
/opt/testing/openmpi/1.3a1r18740/lib/openmpi/mca_pml_ob1.so [0x2a96d66753]
[bru25:11224] [ 4]
/opt/testing/openmpi/1.3a1r18740/lib/openmpi/mca_coll_tuned.so [0x2a97c7cb1c]
[bru25:11224] [ 5]
/opt/testing/openmpi/1.3a1r18740/lib/openmpi/mca_coll_tuned.so [0x2a97c7ce27]
[bru25:11224] [ 6]
/opt/testing/openmpi/1.3a1r18740/lib/openmpi/mca_coll_tuned.so [0x2a97c72eec]
[bru25:11224] [ 7]
/opt/testing/openmpi/1.3a1r18740/lib/libmpi.so.0(PMPI_Bcast+0x13e) [0x2a9559f05e]
[bru25:11224] [ 8] a.out(main+0xd6) [0x400d0f]
[bru25:11224] [ 9] /lib64/tls/libc.so.6(__libc_start_main+0xdb) [0x32ce51c4bb]
[bru25:11224] [10] a.out [0x400b7a]
[bru25:11224] *** End of error message ***
Process 2 of 12 is on bru29
[bru34:29907]
--------------------------------------------------------------------------
mpirun noticed that process rank 0 with PID 11224 on node ic-bru25 exited
on signal 11 (Segmentation fault).
--------------------------------------------------------------------------

Any help much appreciated!
-Doug