Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

From: Jeff Squyres (jsquyres_at_[hidden])
Date: 2005-11-22 22:57:06


On Nov 21, 2005, at 4:44 PM, Enrique Curchitser wrote:

> OK, I'll take you up on the offer. I have 4 Power Mac G5's on a
> private network connected through a GigE switch. Even for large
> problems
> the communications are slugish. This same code has shown to scale
> to upwards of 128 processors on IBM SP's. So here is the output
> to ompi_info --param btl tcp. Any help tunning this, or other
> parameters would be greatly appreciated. Are there any particular
> plugins for TCP?

The TCP BTL is really the only plugin for MPI TCP support in Open MPI.

> MCA btl: parameter "btl_base_debug" (current value: "0")
> If btl_base_debug is 1 standard debug is
> output, if > 1 verbose debug is output
> MCA btl: parameter "btl" (current value: <none>)
> Default selection set of components for
> the btl framework (<none> means "use all components that can be found")
> MCA btl: parameter "btl_base_verbose" (current
> value: "0")
> Verbosity level for the btl framework (0 =
> no verbosity)

These are pretty self-explanitory; they won't help with performance
tuning.

> MCA btl: parameter "btl_tcp_if_include" (current
> value: "en1")
> MCA btl: parameter "btl_tcp_if_exclude" (current
> value: "lo")

These will help you include and exclude various TCP interfaces on your
machine. So they're not really for tuning so much as selecting which
network you want to run over. They're pretty much only useful on
machines with multiple TCP networks.

> MCA btl: parameter "btl_tcp_free_list_num" (current
> value: "8")
> MCA btl: parameter "btl_tcp_free_list_max" (current
> value: "-1")
> MCA btl: parameter "btl_tcp_free_list_inc" (current
> value: "32")

This is a free list of buffers for copying non-contiguous user data
into on a send. That is, if you have an MPI_Datatype that represents
non-contiguous data, we'll first copy it into buffers from this
freelist and then send from there. We start with 8 initial buffers and
grow the list 32 buffers at a time (-1 mean no max number of buffers).
This probably shouldn't have too much to do with performance.

> MCA btl: parameter "btl_tcp_sndbuf" (current value:
> "131072")
> MCA btl: parameter "btl_tcp_rcvbuf" (current value:
> "131072")

This is the OS socket buffer sizing (i.e., how much internal buffer
space the OS has for each socket, in bytes). I have a dim recollection
that Linux has a max size for this, and if you set it above Linux's max
size, it's silently/implicitly downsized to the max. So increasing the
value arbitrarily doesn't help much.

> MCA btl: parameter "btl_tcp_exclusivity" (current
> value: "0")

This has to do with the priority of the tcp btl relative to the other
btl's. Right now, it's set to the lowest value so that tcp is always
chosen after all the rest (e.g., GM, IB, and shared memory all have
higher exclusivity values).

> MCA btl: parameter "btl_tcp_eager_limit" (current
> value: "65536")

This is the largest size that we'll send in the first fragment
(eagerly). If you have a fast network, you might try increasing this.

> MCA btl: parameter "btl_tcp_min_send_size" (current
> value: "65536")
> MCA btl: parameter "btl_tcp_max_send_size" (current
> value: "131072")
> MCA btl: parameter "btl_tcp_min_rdma_size" (current
> value: "131072")
> MCA btl: parameter "btl_tcp_max_rdma_size" (current
> value: "2147483647")
> MCA btl: parameter "btl_tcp_flags" (current value:
> "10")

I'm going to defer explanation of these params to Galen. Galen?

> MCA btl: parameter "btl_tcp_priority" (current
> value: "0")

This parameter is not actually used.

-- 
{+} Jeff Squyres
{+} The Open MPI Project
{+} http://www.open-mpi.org/