Open MPI logo

Open MPI Development Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Development mailing list

Subject: Re: [OMPI devel] Ticket 1224: disable early completion in v1.2.x series
From: Jeff Squyres (jsquyres_at_[hidden])
Date: 2008-03-03 12:56:54

On Mar 3, 2008, at 12:48 PM, Shipman, Galen M. wrote:

> Unfortunately this adds an "if" to the critical path.
> You should at least use OPAL_UNLIKELY..

I could have sworn there was no OPAL_UNLIKELY in the 1.2 series, which
is why I didn't add it. But I just checked right now and I see that
it's there. Doh! So I'll add 2 UNLIKELY's and one LIKELY to the
patch and amend the ticket (i.e., default to "will probably use early

Before adding the UN/LIKELY's, I ran the following tests:

slightly older hardware (pre-woodcrest), netpipe 3.7.1, 1 byte sends:

1.63us with patch, disabled (use_early_completion==0)
  --> saw lots of 1.6xus and 1.9xus results -- nothing in between

1.54us with patch, enabled (use_early_completion==1)
  --> mostly 1.5x, 1.6x, 1.7xus results -- never 1.8x or higher

Saw about same results with vanilla 1.2.5 (no patch) as with
use_early_completion==1 -- in the noise difference.

If someone else could verify these results, it would be great.

> On Mar 3, 2008, at 12:28 PM, Jeff Squyres wrote:
>> The topic of the "early completion" behavior in OB1 for IB
>> optimizations has come up several times in the v1.2 series (it causes
>> problems in some scenarios).
>> - leave the default the way it is now (early completions enabled)
>> - add an MCA parameter for disabling early completions
>> I mention this now because I had a customer complain about it over
>> the
>> weekend. :-)
>> Gleb and I propose the patch in
>> ticket/1224
>> for the v1.2 series. The new OB1 MCA parameter
>> pml_ob1_use_early_completions defaults to 1 (preserving the same
>> behavior as the rest of the v1.2 series), but it can be set to 0 if
>> the early completions on IB are creating problems for specific
>> applications.
>> It would be good to get this functionality in a real release (e.g.,
>> v1.2.6).
>> Note that this MCA parameter is not necessary for the upcoming v1.3
>> series because of changes in ob1 and the openib btl.
>> --
>> Jeff Squyres
>> Cisco Systems
>> _______________________________________________
>> devel mailing list
>> devel_at_[hidden]
> _______________________________________________
> devel mailing list
> devel_at_[hidden]

Jeff Squyres
Cisco Systems