Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

From: Peter Kjellstrom (cap_at_[hidden])
Date: 2007-01-18 08:11:56

On Thursday 18 January 2007 13:08, Scott Atchley wrote:
> The best uni-directional performance I have heard of for PCIe 8x IB
> DDR is ~1,400 MB/s (11.2 Gb/s)

This is on par with what I have seen.

> with Lustre, which is about 55% of the
> theoretical 20 Gb/s advertised speed.

I think this should be calculated against 16 Gbps, not 20 Gbps.

> The ~900 MB/s (7.2 Gb/s)
> mentioned above is, of course, ~72% of advertised speed. If any IB
> folks have any better numbers, please correct me.

Using MPI (over a non idle multi-level switch) I get 940 * 10^6 Bytes/s which
is 94% of peak for that IB 4x SDR.

> The data throughput limit for 8x PCIe is ~12 Gb/s. The theoretical
> limit is 16 Gb/s, but each PCIe packet has a whopping 20 byte
> overhead. If the adapter uses 64 byte packets, then you see 1/3 of
> the throughput go to overhead.

AFAIK the datafield of a pci-express packet is 0-4096 bytes and the header a
bit more than 20 bytes (including things such as start/stop frame bytes,
LCRC/ECRC..). This gives a maximum speed over 4x PCIe of 993.3 10^6 Bytes/s
(8 Gbps after coding minus header waste for a full 4096 byte payload).

In short, the SDR IB equipment I have seen has easily reached 90%+ while
PCI-express on the platforms I've tried has been limited to ~75%. Current IB
DDR HCAs are probably limited by (at least) PCI-express 8x.


  • application/pgp-signature attachment: stored