On Jan 17, 2007, at 2:39 AM, Gleb Natapov wrote:
> Hi Robin,
> On Wed, Jan 17, 2007 at 04:12:10AM -0500, Robin Humble wrote:
>> so this isn't really an OpenMPI questions (I don't think), but you
>> will have hit the problem if anyone has...
>> basically I'm seeing wildly different bandwidths over InfiniBand
>> 4x DDR
>> when I use different kernels.
>> I'm testing with netpipe-3.6.2's NPmpi, but a home-grown pingpong
>> the same thing.
>> the default 2.6.9-42.0.3.ELsmp (and also sles10's kernel) gives ok
>> bandwidth (50% of peak I guess is good?) at ~10 Gbit/s, but a pile of
>> newer kernels (22.214.171.124, 2.6.20-rc4,
>> all max out at ~5.3 Gbit/s.
>> half the bandwidth! :-(
>> latency is the same.
> Try to load ib_mthca with tune_pci=1 option on those kernels that are
I can't speak to the kernels, but one note about bandwidth. By
default, Open MPI uses a pipelined pinning protocol for large message
transfer that provides the best bandwidth when the application has
low buffer reuse and does not require the use of intercepts in the
malloc library or using mallopt to prevent libc from returning memory
to the OS. We have another mode that provides much better bandwidth
when an application has high buffer reuse (like NetPIPE), which can
be enabled by adding "-mca mpi_leave_pinned 1" to the mpirun command
It would be interesting to know if the bandwidth differences appear
when the leave pinned protocol is used. My guess is that they will
not, but one never knows. If so, there are a couple of different
possibilities for why there is the slowdown: higher memory pinning
times, an interaction that throws off our pipeline, etc...
Open MPI Team, CCS-1
Los Alamos National Laboratory