On Mon, 2007-12-17 at 17:19 -0500, Jeff Squyres wrote:
> On Dec 17, 2007, at 8:35 AM, Marco Sbrighi wrote:
> > I'm using Open MPI 1.2.2 over OFED 1.2 on an 256 nodes, dual Opteron,
> > dual core, Linux cluster. Of course, with Infiniband 4x interconnect.
> > Each cluster node is equipped with 4 (or more) ethernet interface,
> > namely 2 gigabit ones plus 2 IPoIB. The two gig are named eth0,eth1,
> > while the two IPoIB are named ib0,ib1.
> > It happens that the eth0 is a management network, with poor
> > performances, and furthermore we wouldn't use the ib* to carry MPI's
> > traffic (neither OOB or TCP), so we would like the eth1 is used for
> > open
> > MPI OOB and TCP.
> > In order to drive the OOB over only eth1 I've tried various
> > combinations
> > of oob_tcp_[ex|in]clude MCA statements, starting from the obvious
> > oob_tcp_exclude = lo,eth0,ib0,ib1
> > then trying the othe obvious:
> > oob_tcp_include = eth1
> This one statement (_include) should be sufficient.
I agree with your interpretation, but what I'm experimenting here is "it
should" but in fact it doesn't .....
> Assumedly this(these) statement(s) are in a config file that is being
> read by Open MPI, such as $HOME/.openmpi/mca-params.conf?
I've tried many combinations: only in $HOME/.openmpi/mca-params.conf,
only in command line and both; but none seems to work correctly.
Nevertheless, what I'm expecting is that if something is specified in
$HOME/.openmpi/mca-params.conf, then if differently specified in command
line, the last should be assumed, I think.
> > and both at the same time.
> > Next I've tried the following:
> > oob_tcp_exclude = eth0
> > but after the job starts, I still have a lot of tcp connections
> > established using eth0 or ib0 or ib1.
> > Furthermore It happens the following error:
> > [node191:03976] [0,1,14]-[0,1,12] mca_oob_tcp_peer_complete_connect:
> > connection failed: Connection timed out (110) - retrying
> This is quite odd. :-(
> > I've found only a way in order to have tcp connections binded only to
> > the eth1 interface, using both the following MCA directives in the
> > command line:
> > mpirun .... --mca oob_tcp_include eth1 --mca oob_tcp_include
> > lo,eth0,ib0,ib1 .....
> > This sounds me as bug.
> Yes, it does. Specifying the MCA same param twice on the command line
> results in undefined behavior -- it will only take one of them, and I
> assume it'll take the first (but I'd have to check the code to be sure).
OK, I can obtain the same behaviour using only one statement:
--mca oob_tcp_include eth1,lo,eth0,ib0,ib1
note that using --mca mpi_show_mca_params what I'm seeing in the report
is the same for both statements (twice and single):
> > Is there someone able to reproduce this behaviour?
> > If this is a bug, are there fixes?
> I'm unfortunately unable to reproduce this behavior. I have a test
> cluster with 2 IP interfaces: ib0, eth0. I have tried several
> combinations of MCA params with 1.2.2:
> --mca oob_tcp_include ib0
> --mca oob_tcp_include ib0,bogus
> --mca oob_tcp_include eth0
> --mca oob_tcp_include eth0,bogus
> --mca oob_tcp_exclude ib0
> --mca oob_tcp_exclude ib0,bogus
> --mca oob_tcp_exclude eth0
> --mca oob_tcp_exclude eth0,bogus
> All do as they are supposed to -- including or excluding ib0 or eth0.
> I do note, however, that the handling of these parameters changed in
> 1.2.3 -- as well as their names. The names changed to
> "oob_tcp_if_include" and "oob_tcp_if_exclude" to match other MCA
> parameter name conventions from other components.
> Could you try with 1.2.3 or 1.2.4 (1.2.4 is the most recent; 1.2.5 is
> due out "soon" -- it *may* get out before the holiday break, but no
we have 1.2.3 in another cluster and it performs the same behaviour as
1.2.2 .... (BTW the other cluster has the same eth ifaces)
> If you can't upgrade, let me know and I can provide a debugging patch
> that will give us a little more insight into what is happening on your
> machines. Thanks.
It is quite difficult for us to upgrade the open-mpi now. We have the
official CISCO packages installed, and I know the 1.2.2-1 is the only
official CISCO's open-mpi distribution today ....
In any case I would like to try your debug patch.
Marco Sbrighi m.sbrighi_at_[hidden]
CINECA Interuniversity Computing Centre
via Magnanelli, 6/3
40033 Casalecchio di Reno (Bo) ITALY
tel. 051 6171516