On Feb 6, 2007, at 12:38 PM, Alex Tumanov wrote:
> The pointer was rather informative. We do have to use non-standard
> ranges for IB interfaces, because we're performing automatic IP over
> IB configuration based on the eth0 IP and netmask. Given 10.x.y.z/8
> configuration for eth0, the IPs assigned to infiniband interfaces will
> not only end up on the same subnet ID, but may even conflict with
> existing ethernet interface IP addresses. Hence the use of 20.x.y.z
> and 30.x.y.z for ib0 & ib1 respectively.
I'm not sure I'm parsing your explanation properly. Are you saying
that your cluster's ethernet addresses are dispersed across all of
10.x.y.z, and therefore you don't want the IPoIB addresses to
conflict? Even being conservative, that's 250^3 IP addresses (over
15 million). There should be plenty of space in there for your
cluster's ethernet and IPoIB addresses to share (and any other
machines that also share your 10.x.y.z address space).
But it doesn't really matter -- this is a minor point. :-)
> I actually tried benchmarking with HPLinpack. Specifically, I'm
> interested in measuring performance improvements when running OpenMPI
> jobs over several available interconnects. However, I have difficulty
> interpreting the cryptic HPL output. I've seen members of the list
> using xhpl benchmark. Perhaps someone could shed some light on how to
> read its output? Also, my understanding is that the only advantage of
I'll defer to others on this one...
> multiple interconnect availability is the increased bandwidth for
> OpenMPI message striping - correct? In that case, I would probably
That's a big reason, yes.
> benefit from a more bandwidth intensive benchmark. If the OpenMPI
> community could point me in the right direction for that, it would be
> greatly appreciated. I have a feeling that this is not one of HPL's
> strongest points.
Actually, it depends on how big your HPL problem size it. HPL can
send very large messages if you set the size high enough. For
example, when we were running HPL at Sandia for its Top500 run, we
were seeing 800MB messages (for 4000+ nodes, lotsa memory -- very
large HPL problem size).
A simple ping-pong benchmark can also be useful to ballpark what
you're seeing for your network performance. My personal favorite is
NetPIPE, but there's others as well.
Server Virtualization Business Unit