Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: Re: [OMPI users] TCP Latency
From: Jeff Squyres (jsquyres_at_[hidden])
Date: 2008-07-29 07:57:37

On Jul 29, 2008, at 6:01 AM, George Bosilca wrote:

> I just want to make sure that I correctly understand your statement.
> You're saying that running NetPIPE (NPtcp) directly over TCP give
> you a latency of 12us, but running NetPIPE (NPmpi) over Open MPI
> bring this latency up to 45us ?

That does sound pretty fishy.

IIRC, Chelsio NICs have two modes of operation: "normal" and offload.
Are you using the normal mode of operation (i.e., go through the
normal Linux kernel TCP stack, etc. -- just like any other TCP/
ethernet NIC) or are you using the offload mode, where OS bypass is
used and all TCP processing is done on the NIC itself? I have not
done any testing myself, but I suspect that there could be a large
difference in latency between the two.

Jeff Squyres
Cisco Systems