When I ran multiple processes in a single machine, the programs are hanging in mpi_allreduce in different points
during different runs. I am using 1.3.4. When I used different machines to run the processes, it is OK. Also, when
I recompiled open mpi in debug mode, the problem goes away. Since the hangings occurred at different points,
I suspect some race/deadlock situations that due to some optimization in openmpi. I used -O3 in compilation with
gcc44 and gfortran44. The software I am running in MUMPS (4.10.0). Other platforms (solaris 10) do not  have
this problem. Any suggestion I should try out?

Here is stack:

#0  mca_btl_sm_component_progress () at btl_sm_component.c:387
#1  0x00002b304a4e1f3a in opal_progress () at runtime/opal_progress.c:207
#2  0x00002b3049e20fa5 in opal_condition_wait (count=2, requests=0x7fff1376d850, statuses=0x0)
    at ../opal/threads/condition.h:99
#3  ompi_request_default_wait_all (count=2, requests=0x7fff1376d850, statuses=0x0)
    at request/req_wait.c:262
#4  0x00002b304ecb4952 in ompi_coll_tuned_allreduce_intra_recursivedoubling (
    sbuf=<value optimized out>, rbuf=0x14c9da10, count=1, dtype=0x2b304a085d40, op=0x2b304a07d280,
    comm=0x14ca34d0, module=0x14ca0500) at coll_tuned_allreduce.c:223
#5  0x00002b3049e36384 in PMPI_Allreduce (sendbuf=0x14c9d8d0, recvbuf=0x14c9da10, count=1,
    datatype=<value optimized out>, op=0x2b304a07d280, comm=0x14ca34d0) at pallreduce.c:102
#6  0x00002b304a0b9bd3 in mpi_allreduce_f (sendbuf=0x14c9d8d0 "", recvbuf=0x14c9da10 "",
    count=0x626eb0, datatype=<value optimized out>, op=0x626ec0, comm=<value optimized out>,
    ierr=0x7fff1376e530) at pallreduce_f.c:77
#7  0x000000000049dbd4 in dmumps_142 (id=...) at dmumps_part5.F:5570