Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: Re: [OMPI users] OpenMPI without IPoIB
From: Shamis, Pavel (shamisp_at_[hidden])
Date: 2011-03-15 16:10:50

I would recommend you to read OFED (or Mellanox OFED) documentation. It will be good start point.


Pavel (Pasha) Shamis

Application Performance Tools Group
Computer Science and Math Division
Oak Ridge National Laboratory
On Mar 14, 2011, at 4:37 PM, Bernardo F Costa wrote:
> Ok. Native ibverbs/openib is preferable although cannot be used by all
> applications (those who do not have a native ip interface). I suppose
> that if I configure my network nodes to use ipoib (by simply probing
> ib_ipoib module) I'd still be able to use native ibverbs interface
> without any delay caused by ipoib on it. And by this way other
> applications which aren't able to use native ibverbs could use the
> infiniband network as well. It should be the reason why some people
> use ipoib I believe, just to offer infiniband to all network
> applications. The main reason I've asked this question is that I have
> seen lots of references in the net on how to configure an infiniband
> network with ipoib, but I was not able to see much references on doing
> the same without ipoib. This made me believe configuring infiniband
> with ipoib could be a popular option.
> I've tried ibdiagnet and other ofed tools. I also tried to debug the
> network environment with simple jobs to measure bandwidth and latency.
> In most cases, I've seen high peaks of measures who come and go
> without any reason I could catch for now. I believe I should check the
> network configuration and make some tests on it. Does anybody here
> know some reference about configuring inifiband without ipoib and/or
> the issues raised when doing this ? If possible, I'd like to see ways
> of testing the configuration, or know about options that could
> increase fault tolerance. I know this is some kind of basic, but I am
> not a well experienced user on infiniband.
> 2011/3/14  <users-request_at_[hidden]>:
>> Please see my comment below.
>> Pavel (Pasha) Shamis
>> ---
>> Application Performance Tools Group
>> Computer Science and Math Division
>> Oak Ridge National Laboratory
>> On Mar 11, 2011, at 2:47 PM, Bernardo F Costa wrote:
>>> I have found this thread form two years ago. I am some kind of lost on
>>> configuring an infiniband cluster for openmpi. What is best: use iboip
>>> or use native infiniband ibverbs interface ? For now I am using native
>> Native openib/verbs interface will work much faster  (up X10) then ipoib.
>> Ipoib was designed for application that does not have native ip interface/
>>> infiniband withou ipoib. But I have lots of problems specially with
>>> latency in the cluster.
>> If you see latency problems over native interface (verbs), then you apparently will
>> face the same problem with any other application over verbs, including ipoib.
>> So using ipoib instead of verbs definitely not a work around for you.
>> I would suggest you to run IB network debug tools , like ibdiagnet , in order to analyze
>> your network/latency problems.
>> Regards,
>> Pasha
> _______________________________________________
> users mailing list
> users_at_[hidden]