Brad Penoff wrote:
> Any objections to us committing an SCTP BTL to ompi-trunk if it has
> the ompi_ignore file in it first?
I'd like to see this in the trunk, though I'd guess that others will
want to know how you plan to support/maintain this code long-term once
it's in. I don't think an ompi_ignore is necessary either, as long as
your configure checks are right.
Do you have any publications on this work?
> For fault tolerance purposes, SCTP connections (termed "associations")
> can be made aware of multiple interfaces on the endpoints by binding
> to more than one interface (for performance, the CMT extension uses
> this multihoming feature to stripe data). SCTP also has several
> different APIs that it supports. Like TCP, there can be a one-to-one
> socket per connection. Another option is that like UDP, there can be
> a single one-to-many socket that is used for all connections. The
> SCTP BTL has the option of using either socket style, depending on the
> value of the btl_sctp_if_11 MCA option. When this value is 1, the
> one-to-one socket is used and like the TCP BTL, there are as many BTL
> component modules as the number of network cards specified with
> if_include and friends. By default, this value is 0 which means that
> a single one-to-many socket is used; here only one BTL module is used
> and internally, SCTP itself handles within that one socket all the
> network cards specified with if_include, etc.
Sounds like a good setup. Have you done performance/resource
utilization/scaling comparisons of the two approaches, as well as how
they compare to the TCP BTL?
> Currently, both the one-to-one and the one-to-many make use of the
> event library offered by Open MPI. The callback functions for the
> one-to-many style however are quite unique as multiple endpoints may
> be interested in the events that poll returns. Currently we use these
> unique callback functions, but in the future the hope is to play with
> the potential benefits of a btl_progress function, particularly for
> the one-to-many style.
In my experience the event callbacks have a high overhead compared to a
progress function, so I'd say thats definitely worth checking out.