I’m running OpenMPI 1.6.4 and seeing a problem where mpi_waitall never returns.  The case runs fine with MVAPICH.  The logic associated with the communications has been extensively debugged in the past; we don’t think it has errors.   Each process posts non-blocking receives, non-blocking sends, and then does waitall on all the outstanding requests. 


The work is broken down into 960 chunks. If I run with 960 processes (60 nodes of 16 cores each), things seem to work.  If I use 160 processes (each process handling 6 chunks of work), then each process is handling 6 times as much communication, and that is the case that hangs with OpenMPI 1.6.4; again, seems to work with MVAPICH.  Is there an obvious place to start, diagnostically?  We’re using the openib btl.