Hi Jeff,<br><br>Given the situation that 16 nodes have eth0,eth1 and eth2 interface, I tried to run data transfer within themselves using mpirun, but without specifying &quot;btl_tcp_if_include&quot;. I got only 15% increase in uni-directional data transfer rate when using 3 links. But if I run two such processes : one using eth0+eth1 and other using eth2 only, I get expected 50% increase in tranfer rate. Any clue?<br>
<br>Regards,<br>Jayanta<br><br><div class="gmail_quote">On Wed, Sep 2, 2009 at 1:41 PM, Jeff Squyres <span dir="ltr">&lt;<a href="mailto:jsquyres@cisco.com">jsquyres@cisco.com</a>&gt;</span> wrote:<br><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
If you don&#39;t use btl_tcp_if_include, Open MPI should use all available ethernet devices, and *should* (although I haven&#39;t tested this recently) only use devices that are routable to specific peers.  Specifically, if you&#39;re on a node with eth0-3, it should use all of them to connect to another peer that has eth0-3, but only use eth0-1 to connect to a peer that only has those 2 devices.  (all of the above assume that all your eth0&#39;s are on one subnet, all your eth1&#39;s are on another subnet, ...etc.)<br>

<br>
Does that work for you?<div><div></div><div class="h5"><br>
<br>
<br>
On Aug 25, 2009, at 7:14 PM, Jayanta Roy wrote:<br>
<br>
</div></div><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"><div><div></div><div class="h5">
Hi,<br>
<br>
I am using Openmpi (version 1.2.2) for MPI data transfer using non-blocking MPI calls like MPI_Isend, MPI_Irecv etc. I am using &quot;--mca btl_tcp_if_include eth0,eth1&quot; to use both the eth link for data transfer within 48 nodes.  Now I have added eth2 and eth3 links on the 32 compute nodes. My aim is to share the high speed data within 32 compute nodes through eth2 and eth3. But I can&#39;t include this as part of &quot;mca&quot; as the rest of 16 nodes do not have these additional interfaces. In MPI/Openmp can one specify explicit routing table within a set of nodes? Such that I can edit /etc/host for additional hostname with these new interfaces and add these hosts in the mpi hostfile.<br>

<br>
Regards,<br></div></div>
Jayanta _______________________________________________<div class="im"><br>
users mailing list<br>
<a href="mailto:users@open-mpi.org" target="_blank">users@open-mpi.org</a><br>
<a href="http://www.open-mpi.org/mailman/listinfo.cgi/users" target="_blank">http://www.open-mpi.org/mailman/listinfo.cgi/users</a><br>
</div></blockquote><font color="#888888">
<br>
<br>
-- <br>
Jeff Squyres<br>
<a href="mailto:jsquyres@cisco.com" target="_blank">jsquyres@cisco.com</a></font><div><div></div><div class="h5"><br>
<br>
_______________________________________________<br>
users mailing list<br>
<a href="mailto:users@open-mpi.org" target="_blank">users@open-mpi.org</a><br>
<a href="http://www.open-mpi.org/mailman/listinfo.cgi/users" target="_blank">http://www.open-mpi.org/mailman/listinfo.cgi/users</a><br>
</div></div></blockquote></div><br>