On Mar 13, 2006, at 8:38 AM, Michael Kluskens wrote:
> On Mar 11, 2006, at 1:00 PM, Jayabrata Chakrabarty wrote:
>> Hi I have been looking for information on how to use multiple
>> Gigabit Ethernet Interface for MPI communication.
>> So far what i have found out is i have to use mca_btl_tcp.
>> But what i wish to know, is what IP Address to assign to each
>> Network Interface. I also wish to know if there will be any change
>> in the format of "hostfile"
>> I have two Gigabit Ethernet Interface on a cluster of 5 nodes at
> It seems to me that an easier approach would be to bond the ethernet
> interfaces together at the Unix/Linux level and then you have only
> one ethernet interface to worry about in MPI. Our Operton-based
> cluster shipped with that setup in SUSE Linux. When I rebuilt it
> with Debian Linux I configured the ethernet interface bonding myself
> using references I found via google. My master node has three
> physical interfaces and two ip addresses, all the rest have two
> physical interfaces and one ip address.
> I have not tested throughput to see if I choose the best type of
> bonding, but the choices were clear enough.
That is one option, yes. However, Channel bonding can result in much
lower performance than letting Open MPI do the stripping and
fragmenting. This is true for a couple of reasons. Channel bonding
requires that packet delivery be in order, so it can not round-robin
short message delivery. While we may have to queue a message
temporarily, we can effectively use both NICs for short messages.
Second, our effective bandwidth for large messages should be nearly N
* effective bandwidth for one NIC. This is rarely the case for
channel bonding, again because of ordering issues. We don't even
have to queue long message fragments internally in the multi-nic
case, as we can immediately write that part of the message into user
space (even if it's after a fragment we haven't received yet).
Of course, if you also need more bandwidth for NFS or MPI
implementations that don't support multi-nic usage, you might not
have an option outside of channel bonding.
Open MPI developer