Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: [OMPI users] Highly variable performance
From: Jed Brown (jed_at_[hidden])
Date: 2010-06-02 16:36:39


I'm investigating some very large performance variation and have reduced
the issue to a very simple MPI_Allreduce benchmark. The variability
does not occur for serial jobs, but it does occur within single nodes.
I'm not at all convinced that this is an Open MPI-specific issue (in
fact the same variance is observed with MVAPICH2 which is an available,
but not "recommended", implementation on that cluster) but perhaps
someone here can suggest steps to track down the issue.

The nodes of interest are 4-socket Opteron 8380 (quad core, 2.5 GHz), connected
with QDR InfiniBand. The benchmark loops over

  MPI_Allgather(localdata,nlocal,MPI_DOUBLE,globaldata,nlocal,MPI_DOUBLE,MPI_COMM_WORLD);

with nlocal=10000 (80 KiB messages) 10000 times, so it normally runs in
a few seconds. Open MPI 1.4.1 was compiled with gcc-4.3.3, and this
code was built with mpicc -O2. All submissions were 8 process, timing
and host results are presented below in chronological order. The jobs
were run with 2-minute time limits (to get through the queue easily)
jobs are marked "killed" if they went over this amount of time. Jobs
were usually submitted in batches of 4. The scheduler is LSF-7.0.

The HOST field indicates the node that was actually used, a6* nodes are
of the type described above, a2* nodes are much older (2-socket Opteron
2220 (dual core, 2.8 GHz)) and use a Quadrics network, the timings are
very reliable on these older nodes. When the issue first came up, I was
inclined to blame memory bandwidith issues with other jobs, but the
variance is still visible when our job runs on exactly a full node,
present regardless of affinity settings, and events that don't require
communication are well-balanced in both small and large runs.

I then suspected possible contention between transport layers, ompi_info
gives

  MCA btl: parameter "btl" (current value: "self,sm,openib,tcp", data source: environment)

so the timings below show many variations of restricting these values.
Unfortunately, the variance is large for all combinations, but I find it
notable that -mca btl self,openib is reliably much slower than self,tcp.

Note that some nodes are used in multiple runs, yet there is no strict
relationship where some nodes are "fast", for instance, a6200 is very
slow (6x and more) in the first set, then normal on the subsequent test.
Nevertheless, when the same node appears in temporally nearby tests,
there seems to be a correlation (though there is certainly not enough
data here to establish that with confidence).

As a final observation, I think the performance in all cases is
unreasonably low since the same test on a (unrelated to the cluster)
2-socket Opteron 2356 (quad core, 2.3 GHz) always takes between 9.75 and
10.0 seconds, 30% faster than the fastest observations on the cluster
nodes with faster cores and memory.

# JOB TIME (s) HOST

ompirun
lsf.o240562 killed 8*a6200
lsf.o240563 9.2110e+01 8*a6200
lsf.o240564 1.5638e+01 8*a6237
lsf.o240565 1.3873e+01 8*a6228

ompirun -mca btl self,sm
lsf.o240574 1.6916e+01 8*a6237
lsf.o240575 1.7456e+01 8*a6200
lsf.o240576 1.4183e+01 8*a6161
lsf.o240577 1.3254e+01 8*a6203
lsf.o240578 1.8848e+01 8*a6274

prun (quadrics)
lsf.o240602 1.6168e+01 4*a2108+4*a2109
lsf.o240603 1.6746e+01 4*a2110+4*a2111
lsf.o240604 1.6371e+01 4*a2108+4*a2109
lsf.o240606 1.6867e+01 4*a2110+4*a2111

ompirun -mca btl self,openib
lsf.o240776 3.1463e+01 8*a6203
lsf.o240777 3.0418e+01 8*a6264
lsf.o240778 3.1394e+01 8*a6203
lsf.o240779 3.5111e+01 8*a6274

ompirun -mca self,sm,openib
lsf.o240851 1.3848e+01 8*a6244
lsf.o240852 1.7362e+01 8*a6237
lsf.o240854 1.3266e+01 8*a6204
lsf.o240855 1.3423e+01 8*a6276

ompirun
lsf.o240858 1.4415e+01 8*a6244
lsf.o240859 1.5092e+01 8*a6237
lsf.o240860 1.3940e+01 8*a6204
lsf.o240861 1.5521e+01 8*a6276
lsf.o240903 1.3273e+01 8*a6234
lsf.o240904 1.6700e+01 8*a6206
lsf.o240905 1.4636e+01 8*a6269
lsf.o240906 1.5056e+01 8*a6234

ompirun -mca self,tcp
lsf.o240948 1.8504e+01 8*a6234
lsf.o240949 1.9317e+01 8*a6207
lsf.o240950 1.8964e+01 8*a6234
lsf.o240951 2.0764e+01 8*a6207

ompirun -mca btl self,sm,openib
lsf.o240998 1.3265e+01 8*a6269
lsf.o240999 1.2884e+01 8*a6269
lsf.o241000 1.3092e+01 8*a6234
lsf.o241001 1.3044e+01 8*a6269

ompirun -mca btl self,openib
lsf.o241013 3.1572e+01 8*a6229
lsf.o241014 3.0552e+01 8*a6234
lsf.o241015 3.1813e+01 8*a6229
lsf.o241016 3.2514e+01 8*a6252

ompirun -mca btl self,sm
lsf.o241044 1.3417e+01 8*a6234
lsf.o241045 killed 8*a6232
lsf.o241046 1.4626e+01 8*a6269
lsf.o241047 1.5060e+01 8*a6253
lsf.o241166 1.3179e+01 8*a6228
lsf.o241167 2.7759e+01 8*a6232
lsf.o241168 1.4224e+01 8*a6234
lsf.o241169 1.4825e+01 8*a6228
lsf.o241446 1.4896e+01 8*a6204
lsf.o241447 1.4960e+01 8*a6228
lsf.o241448 1.7622e+01 8*a6222
lsf.o241449 1.5112e+01 8*a6204

ompirun -mca btl self,tcp
lsf.o241556 1.9135e+01 8*a6204
lsf.o241557 2.4365e+01 8*a6261
lsf.o241558 4.2682e+01 8*a6214
lsf.o241560 2.0481e+01 8*a6262

ompirun -mca btl self,sm,openib
lsf.o241635 1.4234e+01 8*a6204
lsf.o241636 1.2024e+01 8*a6214
lsf.o241637 1.2773e+01 8*a6214
lsf.o241638 killed 8*a6214
lsf.o241684 1.8050e+01 8*a6261
lsf.o241686 1.3567e+01 8*a6203
lsf.o241687 1.5020e+01 8*a6228
lsf.o241688 2.2387e+01 8*a6225

ompirun -mca btl self,openib
lsf.o241723 3.0060e+01 8*a6228
lsf.o241724 3.4366e+01 8*a6244
lsf.o241725 3.0033e+01 8*a6203
lsf.o241726 3.0499e+01 8*a6228
lsf.o241741 3.0483e+01 8*a6234
lsf.o241743 6.9527e+01 8*a6225
lsf.o241744 3.0945e+01 8*a6244
lsf.o241745 3.2120e+01 8*a6220

mvapich2 1.4.1
lsf.o243902 1.3661e+01 8*a6243
lsf.o244832 2.9471e+01 8*a6250
lsf.o244833 2.8425e+01 8*a6250
lsf.o244835 1.3644e+01 8*a6261
lsf.o244837 1.3793e+01 8*a6244
lsf.o244838 2.6907e+01 8*a6250
lsf.o247496 1.3632e+01 8*a6244
lsf.o247497 1.3368e+01 8*a6244
lsf.o247499 1.4120e+01 8*a6252

Any suggestions?

Jed