Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

From: Greg Lindahl (greg.lindahl_at_[hidden])
Date: 2006-12-02 17:30:45

On Sat, Dec 02, 2006 at 10:31:30AM -0500, Jeff Squyres wrote:

> FWIW, especially on NUMA machines (like AMDs), physical access to
> network resources (such as NICs / HCAs) can be much faster on
> specific sockets.

Yes, the penalty is actually 50 ns per hop, and you pay it on both
sides. So our 1.28 usec latency rises to 1.38 if you are on the other
socket on both ends, using a 2 socket motherboard.

In our MPI we also have code that pays attention to the NUMA topology
if you have 2 of our HTX HCAs in a system.

-- greg