Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

From: Michael John Hanby (mhanby_at_[hidden])
Date: 2006-12-21 09:53:27

Thanks Jeff and Andrew, those responses were well thought out and very

I'll run the same test explicitly using TCP for a couple runs and then
mvapi a couple times. The results should give the users a warm fuzzy
that OpenMPI is in fact using the expensive faster network (rather than
cheep gig-e).

Thanks again.

-----Original Message-----
From: users-bounces_at_[hidden] [mailto:users-bounces_at_[hidden]] On
Behalf Of Jeff Squyres
Sent: Wednesday, December 20, 2006 10:01 PM
To: Jeff Squyres
Cc: Open MPI Users
Subject: Re: [OMPI users] Infiniband - Any suggestions on "How can you
proveto me that OpenMPI is using it?"

On Dec 20, 2006, at 7:04 PM, Jeff Squyres wrote:

>> I've been asked by the owner of the cluster "How can you prove to me
>> that this openmpi job is using the Infiniband network?"
>> At first I thought a simple netstat -an on the compute nodes might
>> tell
>> me, however I don't see the Infiniband IP's in the list so I'm
>> thinking
>> maybe I need to be looking elsewhere.
> That's correct.

I completely forgot to explain this comment, sorry.

Open MPI does not use TCP over IB -- it uses the native "verbs"
interface (i.e., a lower lever than TCP over IB), and therefore a)
doesn't show up in tcpdump/netstat/etc., and b) gets much better
performance (TCP demands much processing on the host CPU, whereas
verbs can be offloaded to the processor on the HCA).

Jeff Squyres
Server Virtualization Business Unit
Cisco Systems
users mailing list