Open MPI logo

Open MPI Development Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Development mailing list

Subject: Re: [OMPI devel] RFC: PML/CM priority
From: Scott Atchley (atchley_at_[hidden])
Date: 2009-08-11 15:50:43


George,

When asked about MTL versus BTL, we always suggest that users try both
with their application and determine which is best. I have had
customers report the BTL is better on Solaris (memory registration is
expensive and the BTL can overlap registration and communication when
it fragments a large message) and sometimes better on Linux, but not
always.

The most common issue lately is that users see a failure on high core
count machines (8 or 16) due to the fact that both the MTL and BTL are
opening endpoints. They run into the max number of allowable endpoints
and OMPI aborts. I would suggest that OMPI clearly selects one CM and
only open endpoints for that CM, if possible.

Scott

On Aug 11, 2009, at 3:29 PM, George Bosilca wrote:

> Here is an alternative solution. If instead of setting a hard coded
> value for the priority of CM, we make it use the priority of the MTL
> that get selected, we can solve this problem on a case by case
> approach by carefully setting the MTL's priority (bump up the
> portals and PSM one and decrease the MX MTL). As a result we can
> remove all the extra selection logic and priority management from
> the pml_cm_component.c, and still have a satisfactory solution for
> everybody.
>
> george.
>
> On Aug 11, 2009, at 15:23 , Brian W. Barrett wrote:
>
>> On Tue, 11 Aug 2009, Rainer Keller wrote:
>>
>>> When compiling on systems with MX or Portals, we offer MTLs and
>>> BTLs.
>>> If MTLs are used, the PML/CM is loaded as well as the PML/OB1.
>>>
>>>
>>> Question 1: Is favoring OB1 over CM required for any MTL (MX,
>>> Portals, PSM)?
>>
>> George has in the past had srtong feelings on this issue, believing
>> that for MX, OB1 is prefered over CM. For Portals, it's probably
>> in the noise, but the BTL had been better tested than the MTL, so
>> it was left as the default. Obviously, PSM is a much better choice
>> on InfiniPath than straight OFED, hence the odd priority bump.
>>
>> At this point, I would have no objection to making CM's priority
>> higher for Portals.
>>
>>> Question 2: If it is, I would like to reflect this in the default
>>> priorities,
>>> aka have CM have a priority lower than OB1 and in the case of PSM
>>> raising it.
>>
>> I don't have strong feelings on this one.
>>
>> Brian
>> _______________________________________________
>> devel mailing list
>> devel_at_[hidden]
>> http://www.open-mpi.org/mailman/listinfo.cgi/devel
>
> _______________________________________________
> devel mailing list
> devel_at_[hidden]
> http://www.open-mpi.org/mailman/listinfo.cgi/devel