On Sep 21, 2011, at 3:17 PM, Sébastien Boisvert wrote:
> Meanwhile, I contacted some people at SciNet, which is also part of Compute Canada.
> They told me to try Open-MPI 1.4.3 with the Intel compiler with --mca btl self,ofud to use the ofud BTL instead of openib for OpenFabrics transport.
> This worked quite good -- I got a low latency of 35 microseconds. Yay !
That's still pretty terrible.
Per your comments below, yes, ofud was never finished. I believe it doesn't have retransmission code in there, so if anything is dropped by the network (which, in a congested/busy network, there will be drops), the job will likely hang.
The ofud and openib BTLs should have similar latencies. Indeed, openib should actually have slightly lower HRT ping-pong latencies because of protocol and transport differences between the two.
The openib BTL should give about the same latency as the ibv_rc_pingpong, which you cited at about 11 microseconds (I assume there must be multiple hops in that IB network to be that high), which jives with your "only 1 process sends" RAY network test (http://pastebin.com/dWMXsHpa).
It's not uncommon for latency to go up if multiple processes are all banging on the HCA, but it shouldn't go up noticeably if there's only 2 processes on each node doing simple ping-pong tests, for example.
What happens if you run 2 ibv_rc_pingpong's on each node? Or N ibv_rc_pingpongs?
For corporate legal information go to: