Some time ago Mellanox proposed design that should improve current support for heterogeneous clusters (see Design.txt).The design was accepted by IB vendors, and now we propose patch that adds a heterogeneous cluster support. The path leaves one issue that we do not resolve completely. If 2 different procs have different QPs configuration (P/S/X) we print nice warning message that describes that such configuration is not supported and it propose way to resolve the issue.  Theoretically it will be best to provide solution that automatically will resolve the problem, but it will require significant changes on openib blt that we don’t want to introduce in this stage.

Please comment.