Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

From: Andrew J Caird (acaird_at_[hidden])
Date: 2006-12-20 21:50:38

You can also usually watch the counters on the IB cards and
Ethernet cards. For programs that have a lot of communication
between nodes it is quickly obvious which network you're using.

The IB card monitoring is driver specific, but you should have
some tools for this. For Ethernet you can do something like:

last=`/sbin/ifconfig eth1 | grep RX\ pack | awk '{print $2}' |
awk -F: '{print $2}'`; for m in `seq 1 20` ; do sleep 5;
this=`/sbin/ifconfig eth1 | grep RX\ pack | awk '{print $2}' |
awk -F: '{print $2}'`; echo `expr $this - $last` ; last=$this ;

That should all be on one line.


On Wed, 20 Dec 2006, Jeff Squyres wrote:

> On Dec 20, 2006, at 6:28 PM, Michael John Hanby wrote:
>> Howdy, I'm new to cluster administration, MPI and high speed networks.
>> I've compiled my OpenMPI using these settings:
>> ./configure CC='icc' CXX='icpc' FC='ifort' F77='ifort'
>> --with-mvapi=/usr/local/topspin
>> --with-mvapi-libdir=/usr/local/topspin/lib64 --enable-static
>> --prefix=/share/apps/openmpi/1.1.2
> Looks good.
>> I'm running a Gromacs -np 16 job that I submitting using Sun Grid
>> Engine
>> and OpenMPI that's going to run for several hours.
>> The job was submitted with:
>> mpirun -np 16 -machinefile machines mdrun ......
>> I've been asked by the owner of the cluster "How can you prove to me
>> that this openmpi job is using the Infiniband network?"
>> At first I thought a simple netstat -an on the compute nodes might
>> tell
>> me, however I don't see the Infiniband IP's in the list so I'm
>> thinking
>> maybe I need to be looking elsewhere.
> That's correct.
>> Ompi_info reports:
>> MCA mpool: mvapi (MCA v1.0, API v1.0, Component v1.1.2)
>> MCA btl: mvapi (MCA v1.0, API v1.0, Component v1.1.2)
> Good -- this shows that OMPI was properly compiled with MVAPI support
> (have you considered moving to the OpenFabrics/OFED IB stack,
> perchance? See this web page for more details:
>, in particular,
> The usual answer is that you should be able to tell via performance
> that OMPI is using IB. You can do some small runs with an MPI
> network benchmark application (e.g., NetPIPE) to verify this. Runs
> over the IB network will exhibit much better performance than over
> the TCP network.
> In general, Open MPI checks out what networks are available at run
> time and chooses the "best" one(s) to use for MPI traffic. So if it
> sees an IB network, it should automatically use it. Additionally, if
> OMPI has support for the IB network compiled in (e.g., the mvapi
> components) and it *doesn't* find a valid IB network to use at run
> time, the mvapi component will complain and tell you that you're
> likely to get less performance than you expect and then fail over to
> tcp.
> So -- no news is good news. :-)
> Admittedly, some users advocate that an option to mpirun showing
> which networks are being used would be nicer, such as:
> shell$ mpirun --show-me-which-network-i'm-using ...
> But we haven't gotten around to implementing that because the users
> that we [informally] polled said "if you tell me that I'm *not* using
> the high speed network, that's good enough." So that's what we
> implemented first and never got a round tuit to implement the notices
> viewed from the other way around (so to speak). It's on the to-do
> list; it's just low priority. Patches would be gratefully
> accepted. ;-)
> All that being said, you can *force* using the IB network with the
> following (also see
> category=openfabrics#ib-btl):
> shell$ mpirun --mca btl mvapi,self,sm ...
> This tells Open MPI to use the "BTL" (read: MPI point-to-point
> network) using the mvapi, sm (shared memory), and self components
> ("self" is what is used when an MPI application sends to itself).
> Similarly, you can force using TCP with:
> shell$ mpirun --mca btl tcp,self,sm ...
> If you specify the "btl" MCA parameter, OMPI will only use the
> components that you tell it to use. Or, you can tell OMPI which
> components *not* to use, such as:
> shell$ mpirun --mca btl ^tcp ...
> Which tells OMPI that it can use any btl component *except* tcp
> (i.e., mvapi, sm, self, ...etc.).
> Hope this helps!
> --
> Jeff Squyres
> Server Virtualization Business Unit
> Cisco Systems
> _______________________________________________
> users mailing list
> users_at_[hidden]