I am currently experimenting with OpenMPI in a multi-cluster setting
where each cluster has its private Myri-10G/MX network besides TCP.
Somehow I was under the assumption that OpenMPI would dynamically find
out the details of this configuration, and use MX where possible
(i.e., intra cluster), and TCP elsewhere.
But from some initial testing it appears OpenMPI-1.2.1 assumes global
connectivity over MX when every particpating host supports MX.
I see MX rather than tcp-level connections between clusters being
tried, which across clusters fails in mx_connect/mx_isend (at the
moment there is no inter-cluster support in MX itself). Besides "mx",
I do include "tcp" in the network option lists of course.
Is this just something that is not yet supported in the current
release, or does it work by providing some extra parameters?
I have not started digging in the code yet.