On Thursday 18 January 2007 13:08, Scott Atchley wrote:
...
> The best uni-directional performance I have heard of for PCIe 8x IB
> DDR is ~1,400 MB/s (11.2 Gb/s)
This is on par with what I have seen.
> with Lustre, which is about 55% of the
> theoretical 20 Gb/s advertised speed.
I think this should be calculated against 16 Gbps, not 20 Gbps.
> The ~900 MB/s (7.2 Gb/s)
> mentioned above is, of course, ~72% of advertised speed. If any IB
> folks have any better numbers, please correct me.
Using MPI (over a non idle multi-level switch) I get 940 * 10^6 Bytes/s which
is 94% of peak for that IB 4x SDR.
> The data throughput limit for 8x PCIe is ~12 Gb/s. The theoretical
> limit is 16 Gb/s, but each PCIe packet has a whopping 20 byte
> overhead. If the adapter uses 64 byte packets, then you see 1/3 of
> the throughput go to overhead.
AFAIK the datafield of a pci-express packet is 0-4096 bytes and the header a
bit more than 20 bytes (including things such as start/stop frame bytes,
LCRC/ECRC..). This gives a maximum speed over 4x PCIe of 993.3 10^6 Bytes/s
(8 Gbps after coding minus header waste for a full 4096 byte payload).
In short, the SDR IB equipment I have seen has easily reached 90%+ while
PCI-express on the platforms I've tried has been limited to ~75%. Current IB
DDR HCAs are probably limited by (at least) PCI-express 8x.
/Peter
- application/pgp-signature attachment: stored
|
![[Flag]](../../../../../community/mirrors/flag-us.gif){kind=small}