Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

From: Gleb Natapov (glebn_at_[hidden])
Date: 2007-01-18 04:08:04


On Thu, Jan 18, 2007 at 03:52:19AM -0500, Robin Humble wrote:
> On Wed, Jan 17, 2007 at 08:55:31AM -0700, Brian W. Barrett wrote:
> >On Jan 17, 2007, at 2:39 AM, Gleb Natapov wrote:
> >> On Wed, Jan 17, 2007 at 04:12:10AM -0500, Robin Humble wrote:
> >>> basically I'm seeing wildly different bandwidths over InfiniBand 4x DDR
> >>> when I use different kernels.
> >> Try to load ib_mthca with tune_pci=1 option on those kernels that are
> >> slow.
> >when an application has high buffer reuse (like NetPIPE), which can
> >be enabled by adding "-mca mpi_leave_pinned 1" to the mpirun command
> >line.
>
> thanks! :-)
> tune_pci=1 makes a huge difference at the top end, and
Well this is broken BIOS then. Look here for more explanation:
https://staging.openfabrics.org/svn/openib/gen2/branches/1.1/ofed/docs/mthca_release_notes.txt
search for "tune_pci=1".

> -mca mpi_leave_pinned 1 adds lots of midrange bandwidth.
>
> latencies (~4us) and the low end performance are all unchanged.
>
> see attached for details.
> most curves are for 2.6.19.2 except the last couple (tagged as old)
> which are for 2.6.9-42.0.3.ELsmp and for which tune_pci changes nothing.
>
> why isn't tune_pci=1 the default I wonder?
> files in /sys/module/ib_mthca/ tell me it's off by default in
> 2.6.9-42.0.3.ELsmp, but the results imply that it's on... maybe PCIe
> handling is very different in that kernel.
This is explained in the link above.

>
> is ~10Gbit the best I can expect from 4x DDR IB with MPI?
> some docs @HP suggest up to 16Gbit (data rate) should be possible, and
> I've heard that 13 or 14 has been achieved before. but those might be
> verbs numbers, or maybe horsepower >> 4 cores of 2.66GHz core2 is
> required?
>
> >It would be interesting to know if the bandwidth differences appear
> >when the leave pinned protocol is used. My guess is that they will
>
> yeah, it definitely makes a difference in the 10kB to 10mB range.
> at around 100kB there's 2x the bandwidth when using pinned.
>
> thanks again!
>
> > Brian Barrett
> > Open MPI Team, CCS-1
> > Los Alamos National Laboratory
>
> how's OpenMPI on Cell? :)
>
> cheers,
> robin
> _______________________________________________
> users mailing list
> users_at_[hidden]
> http://www.open-mpi.org/mailman/listinfo.cgi/users

--
			Gleb.