On Jan 8, 2008, at 7:45 AM, Lenny Verkhovsky wrote:
>> Hence, if HAVE_DECL_AF_INET_SDP==1 and using AF_INET_SDP fails to
>> peer, it might be desirable to try to fail over to using
>> AF_INET_something_else. I'm still technically on vacation :-), so I
>> didn't look *too* closely at your patch, but I think you're doing
>> (failing over if AF_INET_SDP doesn't work because of EAFNOSUPPORT),
>> which is good.
> This is actually not implemented yet.
> Supporting failing over requires opening AF_INET sockets in addition
> SDP sockets, this can cause a problem in large clusters.
What I meant was try to open an SDP socket. If it fails because SDP
is not supported / available to that peer, then open a regular
socket. So you should still always have only 1 socket open to a peer
> If one of the machine is not supporting SDP user will get an error.
Well, that's one way to go, but it's certainly less friendly. It
means that the entire MPI job has to support SDP -- including mpirun.
What about clusters that do not have IB on the head node?
>> Perhaps a more general approach would be to [perhaps additionally]
>> provide an MCA param to allow the user to specify the AF_* value?
>> (AF_INET_SDP is a standardized value, right? I.e., will it be the
>> same on all Linux variants [and someday Solaris]?)
> I didn't find any standard on it, it seems to be "randomly" selected
> since the originally it was 26 and changed to 27 due to conflict with
> kernel's defines.
This might make an even stronger case for having an MCA param for it
-- if the AF_INET_SDP value is so broken that it's effectively random,
it may be necessary to override it on some platforms (especially in
light of binary OMPI and OFED distributions that may not match).
>> Patrick's got a good point: is there a reason not to do this?
>> (LD_PRELOAD and the like) Is it problematic with the remote orted's?
> Yes, it's problematic with remote orted's and it not really
> as you might think.
> Since we can't pass environments' variables to the orted's during
I think this depends on your environment. If you're not using rsh
(which you shouldn't be for a large cluster, which is where SDP would
matter most, right?), the resource manager typically copies the
environment out to the cluster nodes. So an LD_PRELOAD value should
be set for the orteds as well.
I agree that it's problematic for rsh, but that might also be solvable
(with some limits; there's only so many characters that we can pass on
the command line -- we did investigate having a wrapper to the orted
at one point to accept environment variables and then launch the
orted, but this was so problematic / klunky that we abandoned the idea).
> we must preload sdp library to each remote environment ( i.e.
> bashrc ) This will cause all applications to use SDP instead of
> Which means you can't choose specific protocol for specific
> either you are using SDP or AF_INET for all.
> SDP also can be loaded with appropriate /usr/local/ofed/etc/
> configuration but a simple user have no access to it usually.
>> Andrew's got a point point here, too -- accelerating the TCP BTL with
>> SDP seems kinda pointless. I'm guessing that you did it because it
>> was just about the same work as was done in the TCP OOB (for which we
>> have no corresponding verbs interface). Is that right?
> Indeed. But it also seems that SDP has lower overhead than VERBS in
Are you referring to the fact that the avail(%) column is lower for
verbs than SDP/IPoIB? That seems like a pretty weird metric for such
small message counts. What exactly does 77.5% of 0 bytes mean?
My $0.02 is that the other columns are more compelling. :-)
> Tests with Sandia's overlapping benchmark
> VERBS results
> msgsize iterations iter_t work_t overhead base_t
> 0 1000 16.892 15.309 1.583 7.029
> 2 1000 16.852 15.332 1.520 7.144
> 4 1000 16.932 15.312 1.620 7.128
> 8 1000 16.985 15.319 1.666 7.182
> 16 1000 16.886 15.297 1.589 7.219
> 32 1000 16.988 15.311 1.677 7.251
> 64 1000 16.944 15.299 1.645 7.457
> SDP results
> 0 1000 134.902 128.089 6.813 54.691
> 2 1000 135.064 128.196 6.868 55.283
> 4 1000 135.031 128.356 6.675 55.039
> 8 1000 130.460 125.908 4.552 52.010
> 16 1000 135.432 128.694 6.738 55.615
> 32 1000 135.228 128.494 6.734 55.627
> 64 1000 135.470 128.540 6.930 56.583
> IPoIB results
> 0 1000 252.953 247.053 5.900 119.977
> 2 1000 253.336 247.285 6.051 121.573
> 4 1000 254.147 247.041 7.106 122.110
> 8 1000 254.613 248.011 6.602 121.840
> 16 1000 255.662 247.952 7.710 124.738
> 32 1000 255.569 248.057 7.512 127.095
> 64 1000 255.867 248.308 7.559 132.858