Open MPI logo

Open MPI Development Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Development mailing list

Subject: Re: [OMPI devel] problem in the ORTE notifier framework
From: Sylvain Jeaugey (sylvain.jeaugey_at_[hidden])
Date: 2009-05-27 08:21:54


I thought an if-then was 1 cycle. I mean, if you don't break the pipeline,
i.e. use likely() or builtin_expect() or something like that to be sure
that the compiler will generate assembly in the right way, it shouldn't be
more than 1 cycle, perhaps less on some architectures like Itanium. But my
multi-architecture view is somewhat limited to x86 and ia64, so I may be
wrong. I'm personally much more sensitive to cache misses which can easily
make the atomic-inc take hundreds of cycles if the event is out of the
cache.

ORTE_NOTIFIER_VERBOSE(api, counter, threshold,...) is also very close to
what we had in mind : a one-line, single call to track events. Good.

We will continue to dig in this direction using the opal-sos branch.
Thanks a lot,
Sylvain

On Wed, 27 May 2009, Ralph Castain wrote:

> While that is a good way of minimizing the impact of the counter, you still have to do an "if-then" to check if the counter
> exceeds the threshold. This "if-then" also has to get executed every time, and generally consumes more than a few cycles.
>
> To be clear: it isn't the output that is the concern. The output only occurs as an exception case, essentially equivalent
> to dealing with an error, so it can be "slow". The concern is with the impact of testing to see if the output needs to be
> generated as this testing occurs every time we transit the code.
>
> I think Jeff and I are probably closer to agreement on design than it might seem, and may be close to what you might also
> have had in mind. Basically, I was thinking of a macro like this:
>
> ORTE_NOTIFIER_VERBOSE(api, counter, threshold,...)
>
> #if WANT_NOTIFIER_VERBOSE
> opal_atomic_increment(counter);
> if (counter > threshold) {
>     orte_notifier.api(.....)
> }
> #endif
>
> You would set the specific thresholds for each situation via MCA params, so this could be tuned to fit specific needs.
> Those who don't want the penalty can just build normally - those who want this level of information can enable it.
>
> We can then see just how much penalty is involved in real world situations. My guess is that it won't be that big, but it's
> hard to know without seeing how frequently we actually insert this code.
>
> Hope that makes sense
> Ralph
>
>
> On Wed, May 27, 2009 at 1:25 AM, Sylvain Jeaugey <sylvain.jeaugey_at_[hidden]> wrote:
> About performance, I may miss something, but our first goal was to track already slow pathes.
>
> We imagined that it could be possible to add at the beginning (or end) of this "bad path" just one line that
> would basically do an atomic inc. So, in terms of CPU cycles, something like 1 for the inc and maybe 1 jump
> before. Are a couple of cycles really an issue in slow pathes (which take at least hundreds of cycles), or do
> you fear out-of-cache memory accesses - or something else ?
>
> As for outputs, they indeed are slow (and can slow down considerably an application if not synchronized), but
> aggregation on the head node should solve our problems. And if not, we can also disable outputs at runtime.
>
> So, in my opinion, no application should notice a difference (unless you tune the framework to output every
> warning).
>
> Sylvain
>
>
> On Tue, 26 May 2009, Jeff Squyres wrote:
>
> Nadia --
>
> Sorry I didn't get to jump in on the other thread earlier.
>
> We have made considerable changes to the notifier framework in a branch to better support "SOS"
> functionality:
>
>  https://www.open-mpi.org/hg/auth/hgwebdir.cgi/jsquyres/opal-sos
>
> Cisco and Indiana U. have been working on this branch for a while.  A description of the SOS stuff is
> here:
>
>  https://svn.open-mpi.org/trac/ompi/wiki/ErrorMessages
>
> As for setting up an external web server with hg, don't bother -- just get an account at bitbucket.org.
>  They're free and allow you to host hg repositories there.  I've used bitbucket to collaborate on code
> before it hits OMPI's SVN trunk with both internal and external OMPI developers.
>
> We can certainly move the opal-sos repo to bitbucket (or branch again off opal-sos to bitbucket --
> whatever makes more sense) to facilitate collaborating with you.
>
> Back on topic...
>
> I'd actually suggest a combination of what has been discussed in the other thread.  The notifier can be
> the mechanism that actually sends the output message, but it doesn't have to be the mechanism that tracks
> the stats and decides when to output a message.  That can be separate logic, and therefore be more
> fine-grained (and potentially even specific to the MPI layer).
>
> The Big Question will how to do this with zero performance impact when it is not being used. This has
> always been the difficult issue when trying to implement any kind of monitoring inside the core OMPI
> performance-sensitive paths.  Even adding individual branches has met with resistance (in
> performance-critical code paths)...
>
>
>
> On May 26, 2009, at 10:59 AM, Nadia Derbey wrote:
>
> Hi,
>
> While having a look at the notifier framework under orte, I noticed that
> the way it is written, the init routine for the selected module cannot
> be called.
>
> Attached is a small patch that fixes this issue.
>
> Regards,
> Nadia
>
> <orte_notifier_fix_select.patch><ATT14046023.txt>
>
>
>
> --
> Jeff Squyres
> Cisco Systems
>
> _______________________________________________
> devel mailing list
> devel_at_[hidden]
> http://www.open-mpi.org/mailman/listinfo.cgi/devel
>
> _______________________________________________
> devel mailing list
> devel_at_[hidden]
> http://www.open-mpi.org/mailman/listinfo.cgi/devel
>
>
>
>