Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: Re: [OMPI users] OpenMPI without IPoIB
From: Shamis, Pavel (shamisp_at_[hidden])
Date: 2011-03-15 16:10:50


I would recommend you to read OFED (or Mellanox OFED) documentation. It will be good start point.

Regards,

Pavel (Pasha) Shamis

---
Application Performance Tools Group
Computer Science and Math Division
Oak Ridge National Laboratory
On Mar 14, 2011, at 4:37 PM, Bernardo F Costa wrote:
> Ok. Native ibverbs/openib is preferable although cannot be used by all
> applications (those who do not have a native ip interface). I suppose
> that if I configure my network nodes to use ipoib (by simply probing
> ib_ipoib module) I'd still be able to use native ibverbs interface
> without any delay caused by ipoib on it. And by this way other
> applications which aren't able to use native ibverbs could use the
> infiniband network as well. It should be the reason why some people
> use ipoib I believe, just to offer infiniband to all network
> applications. The main reason I've asked this question is that I have
> seen lots of references in the net on how to configure an infiniband
> network with ipoib, but I was not able to see much references on doing
> the same without ipoib. This made me believe configuring infiniband
> with ipoib could be a popular option.
> I've tried ibdiagnet and other ofed tools. I also tried to debug the
> network environment with simple jobs to measure bandwidth and latency.
> In most cases, I've seen high peaks of measures who come and go
> without any reason I could catch for now. I believe I should check the
> network configuration and make some tests on it. Does anybody here
> know some reference about configuring inifiband without ipoib and/or
> the issues raised when doing this ? If possible, I'd like to see ways
> of testing the configuration, or know about options that could
> increase fault tolerance. I know this is some kind of basic, but I am
> not a well experienced user on infiniband.
> 
> 2011/3/14  <users-request_at_[hidden]>:
>> 
>> Please see my comment below.
>> 
>> Pavel (Pasha) Shamis
>> ---
>> Application Performance Tools Group
>> Computer Science and Math Division
>> Oak Ridge National Laboratory
>> 
>> 
>> On Mar 11, 2011, at 2:47 PM, Bernardo F Costa wrote:
>> 
>>> I have found this thread form two years ago. I am some kind of lost on
>>> configuring an infiniband cluster for openmpi. What is best: use iboip
>>> or use native infiniband ibverbs interface ? For now I am using native
>> 
>> Native openib/verbs interface will work much faster  (up X10) then ipoib.
>> Ipoib was designed for application that does not have native ip interface/
>> 
>>> infiniband withou ipoib. But I have lots of problems specially with
>>> latency in the cluster.
>> 
>> If you see latency problems over native interface (verbs), then you apparently will
>> face the same problem with any other application over verbs, including ipoib.
>> 
>> So using ipoib instead of verbs definitely not a work around for you.
>> 
>> I would suggest you to run IB network debug tools , like ibdiagnet , in order to analyze
>> your network/latency problems.
>> 
>> Regards,
>> Pasha
> 
> _______________________________________________
> users mailing list
> users_at_[hidden]
> http://www.open-mpi.org/mailman/listinfo.cgi/users