Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: Re: [OMPI users] btl_openib_cpc_include rdmacm questions
From: Brock Palen (brockp_at_[hidden])
Date: 2011-04-22 10:23:18


On Apr 21, 2011, at 6:49 PM, Ralph Castain wrote:

>
> On Apr 21, 2011, at 4:41 PM, Brock Palen wrote:
>
>> Given that part of our cluster is TCP only, openib wouldn't even startup on those hosts
>
> That is correct - it would have no impact on those hosts
>
>> and this would be ignored on hosts with IB adaptors?
>
> Ummm...not sure I understand this one. The param -will- be used on hosts with IB adaptors because that is what it is controlling.
>
> However, it -won't- have any impact on hosts without IB adaptors, which is what I suspect you meant to ask?

Correct typo, Thanks, I am going to add the environment variable to our OpenMPI modules so rdmacm is our default for now, Thanks!

>
>
>>
>> Just checking thanks!
>>
>> Brock Palen
>> www.umich.edu/~brockp
>> Center for Advanced Computing
>> brockp_at_[hidden]
>> (734)936-1985
>>
>>
>>
>> On Apr 21, 2011, at 6:21 PM, Jeff Squyres wrote:
>>
>>> Over IB, I'm not sure there is much of a drawback. It might be slightly slower to establish QP's, but I don't think that matters much.
>>>
>>> Over iWARP, rdmacm can cause connection storms as you scale to thousands of MPI processes.
>>>
>>>
>>> On Apr 20, 2011, at 5:03 PM, Brock Palen wrote:
>>>
>>>> We managed to have another user hit the bug that causes collectives (this time MPI_Bcast() ) to hang on IB that was fixed by setting:
>>>>
>>>> btl_openib_cpc_include rdmacm
>>>>
>>>> My question is if we set this to the default on our system with an environment variable does it introduce any performance or other issues we should be aware of?
>>>>
>>>> Is there a reason we should not use rdmacm?
>>>>
>>>> Thanks!
>>>>
>>>> Brock Palen
>>>> www.umich.edu/~brockp
>>>> Center for Advanced Computing
>>>> brockp_at_[hidden]
>>>> (734)936-1985
>>>>
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> users mailing list
>>>> users_at_[hidden]
>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>
>>>
>>> --
>>> Jeff Squyres
>>> jsquyres_at_[hidden]
>>> For corporate legal information go to:
>>> http://www.cisco.com/web/about/doing_business/legal/cri/
>>>
>>>
>>> _______________________________________________
>>> users mailing list
>>> users_at_[hidden]
>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>
>>>
>>
>>
>> _______________________________________________
>> users mailing list
>> users_at_[hidden]
>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>
>
> _______________________________________________
> users mailing list
> users_at_[hidden]
> http://www.open-mpi.org/mailman/listinfo.cgi/users
>
>