> -----Original Message-----
> From: devel-bounces_at_[hidden] [mailto:devel-bounces_at_[hidden]]
> Behalf Of Jeff Squyres
> Sent: Tuesday, January 08, 2008 4:32 PM
> To: Open MPI Developers
> Subject: Re: [OMPI devel] SDP support for OPEN-MPI
> On Jan 8, 2008, at 7:45 AM, Lenny Verkhovsky wrote:
> >> Hence, if HAVE_DECL_AF_INET_SDP==1 and using AF_INET_SDP fails to
> >> that
> >> peer, it might be desirable to try to fail over to using
> >> AF_INET_something_else. I'm still technically on vacation :-), so
> >> didn't look *too* closely at your patch, but I think you're doing
> >> that
> >> (failing over if AF_INET_SDP doesn't work because of EAFNOSUPPORT),
> >> which is good.
> > This is actually not implemented yet.
> > Supporting failing over requires opening AF_INET sockets in addition
> > to
> > SDP sockets, this can cause a problem in large clusters.
> What I meant was try to open an SDP socket. If it fails because SDP
> is not supported / available to that peer, then open a regular
> socket. So you should still always have only 1 socket open to a peer
> (not 2).
Yes, but since the listener side doesn't know on which socket to expect
a message it will need both sockets to be opened.
> > If one of the machine is not supporting SDP user will get an error.
> Well, that's one way to go, but it's certainly less friendly. It
> means that the entire MPI job has to support SDP -- including mpirun.
> What about clusters that do not have IB on the head node?
They can use OOB over IP sockets and BTL on SDP, it should work.
> >> Perhaps a more general approach would be to [perhaps additionally]
> >> provide an MCA param to allow the user to specify the AF_* value?
> >> (AF_INET_SDP is a standardized value, right? I.e., will it be the
> >> same on all Linux variants [and someday Solaris]?)
> > I didn't find any standard on it, it seems to be "randomly" selected
> > since the originally it was 26 and changed to 27 due to conflict
> > kernel's defines.
> This might make an even stronger case for having an MCA param for it
> -- if the AF_INET_SDP value is so broken that it's effectively random,
> it may be necessary to override it on some platforms (especially in
> light of binary OMPI and OFED distributions that may not match).
If we talking about passing AF_INET_SDP value only then
1. Passing this value as mca parameter will not make any changes to the
2. Hopefully in the future AF_INET_SDP value can be gotten from the
And the value will be configured automatically.
3. If we are talking about AF_INET value in general ( IPv4, IPv6, SDP)
Then by making it constant with mca parameter we are limiting ourselves
for one protocol only without being able to failover or using different
protocols for different needs ( i.e. SDP for OOB and IPv4 for BTL )
> >> Patrick's got a good point: is there a reason not to do this?
> >> (LD_PRELOAD and the like) Is it problematic with the remote
> > Yes, it's problematic with remote orted's and it not really
> > transparent
> > as you might think.
> > Since we can't pass environments' variables to the orted's during
> > runtime
> I think this depends on your environment. If you're not using rsh
> (which you shouldn't be for a large cluster, which is where SDP would
> matter most, right?), the resource manager typically copies the
> environment out to the cluster nodes. So an LD_PRELOAD value should
> be set for the orteds as well.
> I agree that it's problematic for rsh, but that might also be solvable
> (with some limits; there's only so many characters that we can pass on
> the command line -- we did investigate having a wrapper to the orted
> at one point to accept environment variables and then launch the
> orted, but this was so problematic / klunky that we abandoned the
Using LD_PRELOAD will not allow us to use SDP and IP separately, i.e.
SDP for OOB and IP for a BTL.
> > we must preload sdp library to each remote environment ( i.e.
> > bashrc ) This will cause all applications to use SDP instead of
> > AF_INET.
> > Which means you can't choose specific protocol for specific
> > application,
> > either you are using SDP or AF_INET for all.
> > SDP also can be loaded with appropriate /usr/local/ofed/etc/
> > libsdp.conf
> > configuration but a simple user have no access to it usually.
> > _ug/sdp.htm#wp952927)
> >> Andrew's got a point point here, too -- accelerating the TCP BTL
> >> SDP seems kinda pointless. I'm guessing that you did it because it
> >> was just about the same work as was done in the TCP OOB (for which
> >> have no corresponding verbs interface). Is that right?
> > Indeed. But it also seems that SDP has lower overhead than VERBS in
> > some
> > cases.
> Are you referring to the fact that the avail(%) column is lower for
> verbs than SDP/IPoIB? That seems like a pretty weird metric for such
> small message counts. What exactly does 77.5% of 0 bytes mean?
> My $0.02 is that the other columns are more compelling. :-)
> > Tests with Sandia's overlapping benchmark
> > http://www.cs.sandia.gov/smb/overhead.html#mozTocId316713
> > VERBS results
> > msgsize iterations iter_t work_t overhead base_t
> > avail(%)
> > 0 1000 16.892 15.309 1.583 7.029
> > 77.5
> > 2 1000 16.852 15.332 1.520 7.144
> > 78.7
> > 4 1000 16.932 15.312 1.620 7.128
> > 77.3
> > 8 1000 16.985 15.319 1.666 7.182
> > 76.8
> > 16 1000 16.886 15.297 1.589 7.219
> > 78.0
> > 32 1000 16.988 15.311 1.677 7.251
> > 76.9
> > 64 1000 16.944 15.299 1.645 7.457
> > 77.9
> > SDP results
> > 0 1000 134.902 128.089 6.813 54.691
> > 87.5
> > 2 1000 135.064 128.196 6.868 55.283
> > 87.6
> > 4 1000 135.031 128.356 6.675 55.039
> > 87.9
> > 8 1000 130.460 125.908 4.552 52.010
> > 91.2
> > 16 1000 135.432 128.694 6.738 55.615
> > 87.9
> > 32 1000 135.228 128.494 6.734 55.627
> > 87.9
> > 64 1000 135.470 128.540 6.930 56.583
> > 87.8
> > IPoIB results
> > 0 1000 252.953 247.053 5.900 119.977
> > 95.1
> > 2 1000 253.336 247.285 6.051 121.573
> > 95.0
> > 4 1000 254.147 247.041 7.106 122.110
> > 94.2
> > 8 1000 254.613 248.011 6.602 121.840
> > 94.6
> > 16 1000 255.662 247.952 7.710 124.738
> > 93.8
> > 32 1000 255.569 248.057 7.512 127.095
> > 94.1
> > 64 1000 255.867 248.308 7.559 132.858
> > 94.3
> Jeff Squyres
> Cisco Systems
> devel mailing list