Ok. Native ibverbs/openib is preferable although cannot be used by all
applications (those who do not have a native ip interface). I suppose
that if I configure my network nodes to use ipoib (by simply probing
ib_ipoib module) I'd still be able to use native ibverbs interface
without any delay caused by ipoib on it. And by this way other
applications which aren't able to use native ibverbs could use the
infiniband network as well. It should be the reason why some people
use ipoib I believe, just to offer infiniband to all network
applications. The main reason I've asked this question is that I have
seen lots of references in the net on how to configure an infiniband
network with ipoib, but I was not able to see much references on doing
the same without ipoib. This made me believe configuring infiniband
with ipoib could be a popular option.
I've tried ibdiagnet and other ofed tools. I also tried to debug the
network environment with simple jobs to measure bandwidth and latency.
In most cases, I've seen high peaks of measures who come and go
without any reason I could catch for now. I believe I should check the
network configuration and make some tests on it. Does anybody here
know some reference about configuring inifiband without ipoib and/or
the issues raised when doing this ? If possible, I'd like to see ways
of testing the configuration, or know about options that could
increase fault tolerance. I know this is some kind of basic, but I am
not a well experienced user on infiniband.
> Please see my comment below.
> Pavel (Pasha) Shamis
> Application Performance Tools Group
> Computer Science and Math Division
> Oak Ridge National Laboratory
> On Mar 11, 2011, at 2:47 PM, Bernardo F Costa wrote:
>> I have found this thread form two years ago. I am some kind of lost on
>> configuring an infiniband cluster for openmpi. What is best: use iboip
>> or use native infiniband ibverbs interface ? For now I am using native
> Native openib/verbs interface will work much faster (up X10) then ipoib.
> Ipoib was designed for application that does not have native ip interface/
>> infiniband withou ipoib. But I have lots of problems specially with
>> latency in the cluster.
> If you see latency problems over native interface (verbs), then you apparently will
> face the same problem with any other application over verbs, including ipoib.
> So using ipoib instead of verbs definitely not a work around for you.
> I would suggest you to run IB network debug tools , like ibdiagnet , in order to analyze
> your network/latency problems.