I’m having some trouble getting good scaling with OpenMPI 1.6.4 and I don’t know where to start looking. This is an Infiniband FDR network with Sandy Bridge nodes.  I am using affinity (--bind-to-core) but no other options. As the number of cores goes up, the message sizes are typically going down. There seem to be lots of options in the FAQ, and I would welcome any advice on where to start.  All these timings are on a completely empty system except for me.

 

Thanks

 

 

    MPI              # cores   Ave. Rate   Std. Dev. %  # timings   Speedup    Efficiency

================================================================================================

MVAPICH            |   16   |    8.6783  |   0.995 % |       2  |   16.000  |  1.0000

MVAPICH            |   48   |    8.7665  |   1.937 % |       3  |   47.517  |  0.9899

MVAPICH            |   80   |    8.8900  |   2.291 % |       3  |   78.095  |  0.9762

MVAPICH            |  160   |    8.9897  |   2.409 % |       3  |  154.457  |  0.9654

MVAPICH            |  320   |    8.9780  |   2.801 % |       3  |  309.317  |  0.9666

MVAPICH            |  480   |    8.9704  |   2.316 % |       3  |  464.366  |  0.9674

MVAPICH            |  640   |    9.0792  |   1.138 % |       3  |  611.739  |  0.9558

MVAPICH            |  720   |    9.1328  |   1.052 % |       3  |  684.162  |  0.9502

MVAPICH            |  800   |    9.1945  |   0.773 % |       3  |  755.079  |  0.9438

OpenMPI            |   16   |    8.6743  |   2.335 % |       2  |   16.000  |  1.0000

OpenMPI            |   48   |    8.7826  |   1.605 % |       2  |   47.408  |  0.9877

OpenMPI            |   80   |    8.8861  |   0.120 % |       2  |   78.093  |  0.9762

OpenMPI            |  160   |    8.9774  |   0.785 % |       2  |  154.598  |  0.9662

OpenMPI            |  320   |   12.0585  |  16.950 % |       2  |  230.191  |  0.7193

OpenMPI            |  480   |   14.8330  |   1.300 % |       2  |  280.701  |  0.5848

OpenMPI            |  640   |   17.1723  |   2.577 % |       3  |  323.283  |  0.5051

OpenMPI            |  720   |   18.2153  |   2.798 % |       3  |  342.868  |  0.4762

OpenMPI            |  800   |   19.3603  |   2.254 % |       3  |  358.434  |  0.4480