Correct.  20 nodes, 8 cores per dual-socket on each node = 360.

 

From: users-bounces@open-mpi.org [mailto:users-bounces@open-mpi.org] On Behalf Of Ralph Castain
Sent: Sunday, June 09, 2013 6:18 PM
To: Open MPI Users
Subject: Re: [OMPI users] EXTERNAL: Re: Need advice on performance problem

 

So, just to be sure - when you run 320 "cores", you are running across 20 nodes?

 

Just want to ensure we are using "core" the same way - some people confuse cores with hyperthreads.

 

On Jun 9, 2013, at 3:50 PM, "Blosch, Edwin L" <edwin.l.blosch@lmco.com> wrote:



16.  dual-socket Xeon, E5-2670.

 

I am trying a larger model to see if the performance drop-off happens at a different number of cores. 

Also I’m running some intermediate core-count sizes to refine the curve a bit.

I also added mpi_show_mca_params all, and at the same time, btl_openib_use_eager_rdma 1, just to see if that does anything.

 

From: users-bounces@open-mpi.org [mailto:users-bounces@open-mpi.org] On Behalf Of Ralph Castain
Sent: Sunday, June 09, 2013 5:04 PM
To: Open MPI Users
Subject: EXTERNAL: Re: [OMPI users] Need advice on performance problem

 

Looks to me like things are okay thru 160, and then things fall apart after that point. How many cores are on a node?

 

 

On Jun 9, 2013, at 1:59 PM, "Blosch, Edwin L" <edwin.l.blosch@lmco.com> wrote:




I’m having some trouble getting good scaling with OpenMPI 1.6.4 and I don’t know where to start looking. This is an Infiniband FDR network with Sandy Bridge nodes.  I am using affinity (--bind-to-core) but no other options. As the number of cores goes up, the message sizes are typically going down. There seem to be lots of options in the FAQ, and I would welcome any advice on where to start.  All these timings are on a completely empty system except for me.

 

Thanks

 

 

    MPI              # cores   Ave. Rate   Std. Dev. %  # timings   Speedup    Efficiency

================================================================================================

MVAPICH            |   16   |    8.6783  |   0.995 % |       2  |   16.000  |  1.0000

MVAPICH            |   48   |    8.7665  |   1.937 % |       3  |   47.517  |  0.9899

MVAPICH            |   80   |    8.8900  |   2.291 % |       3  |   78.095  |  0.9762

MVAPICH            |  160   |    8.9897  |   2.409 % |       3  |  154.457  |  0.9654

MVAPICH            |  320   |    8.9780  |   2.801 % |       3  |  309.317  |  0.9666

MVAPICH            |  480   |    8.9704  |   2.316 % |       3  |  464.366  |  0.9674

MVAPICH            |  640   |    9.0792  |   1.138 % |       3  |  611.739  |  0.9558

MVAPICH            |  720   |    9.1328  |   1.052 % |       3  |  684.162  |  0.9502

MVAPICH            |  800   |    9.1945  |   0.773 % |       3  |  755.079  |  0.9438

OpenMPI            |   16   |    8.6743  |   2.335 % |       2  |   16.000  |  1.0000

OpenMPI            |   48   |    8.7826  |   1.605 % |       2  |   47.408  |  0.9877

OpenMPI            |   80   |    8.8861  |   0.120 % |       2  |   78.093  |  0.9762

OpenMPI            |  160   |    8.9774  |   0.785 % |       2  |  154.598  |  0.9662

OpenMPI            |  320   |   12.0585  |  16.950 % |       2  |  230.191  |  0.7193

OpenMPI            |  480   |   14.8330  |   1.300 % |       2  |  280.701  |  0.5848

OpenMPI            |  640   |   17.1723  |   2.577 % |       3  |  323.283  |  0.5051

OpenMPI            |  720   |   18.2153  |   2.798 % |       3  |  342.868  |  0.4762

OpenMPI            |  800   |   19.3603  |   2.254 % |       3  |  358.434  |  0.4480

_______________________________________________
users mailing list
users@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users

 

_______________________________________________
users mailing list
users@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users