Thanks Jeff and Andrew, those responses were well thought out and very
I'll run the same test explicitly using TCP for a couple runs and then
mvapi a couple times. The results should give the users a warm fuzzy
that OpenMPI is in fact using the expensive faster network (rather than
From: users-bounces_at_[hidden] [mailto:users-bounces_at_[hidden]] On
Behalf Of Jeff Squyres
Sent: Wednesday, December 20, 2006 10:01 PM
To: Jeff Squyres
Cc: Open MPI Users
Subject: Re: [OMPI users] Infiniband - Any suggestions on "How can you
proveto me that OpenMPI is using it?"
On Dec 20, 2006, at 7:04 PM, Jeff Squyres wrote:
>> I've been asked by the owner of the cluster "How can you prove to me
>> that this openmpi job is using the Infiniband network?"
>> At first I thought a simple netstat -an on the compute nodes might
>> me, however I don't see the Infiniband IP's in the list so I'm
>> maybe I need to be looking elsewhere.
> That's correct.
I completely forgot to explain this comment, sorry.
Open MPI does not use TCP over IB -- it uses the native "verbs"
interface (i.e., a lower lever than TCP over IB), and therefore a)
doesn't show up in tcpdump/netstat/etc., and b) gets much better
performance (TCP demands much processing on the host CPU, whereas
verbs can be offloaded to the processor on the HCA).
Server Virtualization Business Unit
users mailing list