Note that striping doesn't really help you much until data sizes get
large. For example, networks tend to have an elbow in the graph where
the size of the message starts to matter (clearly evident on your
Additionally, you have your network marked as with "hubs" not
"switches" -- if you really do have hubs and not switches, you may run
into serious contention issues if you start loading up the network.
With both of these factors, even though you have 4 links, you likely
aren't going to see much of a performance benefit until you send large
messages (which will be limited by your bus speeds -- can you feed all
4 of your links from a single machine at line rate, or will you be
limited by PCI bus speeds and contention?), and you may run into
secondary performance issues due to contention on your hubs.
On May 28, 2009, at 11:06 PM, shan axida wrote:
> Thank you! Mr. Jeff Squyres,
> I have conducted a simple MPI_Bcast experiment in out cluster.
> The results are shown in the file attached on this e-mail.
> The hostfile is :
> hostname1 slots=4
> hostname2 slots=4
> hostname3 slots=4
> hostname16 slots=4
> As we can see in the figure, it is little faster than single link
> when we use 2,3,4 links between nodes.
> My question is what would be the reason to make almost the same
> performance when we use 2,3,4 links ?
> Thank you!
> From: Jeff Squyres <jsquyres_at_[hidden]>
> To: Open MPI Users <users_at_[hidden]>
> Sent: Wednesday, May 27, 2009 11:28:42 PM
> Subject: Re: [OMPI users] How to use Multiple links with
> Open MPI considers hosts differently than network links.
> So you should only list the actual hostname in the hostfile, with
> slots equal to the number of processors (4 in your case, I think?).
> Once the MPI processes are launched, they each look around on the
> host that they're running and find network paths to each of their
> peers. If they are multiple paths between pairs of peers, Open MPI
> will round-robin stripe messages across each of the links. We don't
> really have an easy setting for each peer pair only using 1 link.
> Indeed, since connectivity is bidirectional, the traffic patterns
> become less obvious if you want MPI_COMM_WORLD rank X to only use
> link Y -- what does that mean to the other 4 MPI processes on the
> other host (with whom you have assumedly assigned their own
> individual links as well)?
> On May 26, 2009, at 12:24 AM, shan axida wrote:
> > Hi everyone,
> > I want to ask how to use multiple links (multiple NICs) with
> > For example, how can I assign a link to each process, if there are
> 4 links
> > and 4 processors on each node in our cluster?
> > Is this a correct way?
> > hostfile:
> > ----------------------
> > host1-eth0 slots=1
> > host1-eth1 slots=1
> > host1-eth2 slots=1
> > host1-eth3 slots=1
> > host2-eth0 slots=1
> > host2-eth1 slots=1
> > host2-eth2 slots=1
> > host2-eth3 slots=1
> > ... ...
> > ... ...
> > host16-eth0 slots=1
> > host16-eth1 slots=1
> > host16-eth2 slots=1
> > host16-eth3 slots=1
> > ------------------------
> > _______________________________________________
> > users mailing list
> > users_at_[hidden]
> > http://www.open-mpi.org/mailman/listinfo.cgi/users
> --Jeff Squyres
> Cisco Systems
> users mailing list
> users mailing list