Open MPI logo

Open MPI Development Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Development mailing list

Subject: Re: [OMPI devel] problem in the ORTE notifier framework
From: Jeff Squyres (jsquyres_at_[hidden])
Date: 2009-05-27 11:12:30


FYI, I moved the opal-sos repo this morning to bitbucket.org:

     http://bitbucket.org/jsquyres/opal-sos/overview/

If you already pulled from the old www.open-mpi.org copy, you can just
edit your .hg/hgrc to set the new bitbucket URL and continue pulling /
etc. (no need to get a new checkout)

On May 27, 2009, at 9:49 AM, Jeff Squyres (jsquyres) wrote:

> This all sounds reasonable.
>
> If these are, indeed, on already-slow code paths, I doubt there will
> be much of an issue.
>
> If you want to talk about this higher bandwidth, let us know -- I can
> setup a Webex call pretty easily.
>
>
> On May 27, 2009, at 8:21 AM, Sylvain Jeaugey wrote:
>
> > I thought an if-then was 1 cycle. I mean, if you don't break the
> > pipeline,
> > i.e. use likely() or builtin_expect() or something like that to be
> > sure
> > that the compiler will generate assembly in the right way, it
> > shouldn't be
> > more than 1 cycle, perhaps less on some architectures like Itanium.
> > But my
> > multi-architecture view is somewhat limited to x86 and ia64, so I
> > may be
> > wrong. I'm personally much more sensitive to cache misses which can
> > easily
> > make the atomic-inc take hundreds of cycles if the event is out of
> the
> > cache.
> >
> > ORTE_NOTIFIER_VERBOSE(api, counter, threshold,...) is also very
> > close to
> > what we had in mind : a one-line, single call to track events. Good.
> >
> > We will continue to dig in this direction using the opal-sos branch.
> > Thanks a lot,
> > Sylvain
> >
> > On Wed, 27 May 2009, Ralph Castain wrote:
> >
> > > While that is a good way of minimizing the impact of the counter,
> > you still have to do an "if-then" to check if the counter
> > > exceeds the threshold. This "if-then" also has to get executed
> > every time, and generally consumes more than a few cycles.
> > >
> > > To be clear: it isn't the output that is the concern. The output
> > only occurs as an exception case, essentially equivalent
> > > to dealing with an error, so it can be "slow". The concern is with
> > the impact of testing to see if the output needs to be
> > > generated as this testing occurs every time we transit the code.
> > >
> > > I think Jeff and I are probably closer to agreement on design than
> > it might seem, and may be close to what you might also
> > > have had in mind. Basically, I was thinking of a macro like this:
> > >
> > > ORTE_NOTIFIER_VERBOSE(api, counter, threshold,...)
> > >
> > > #if WANT_NOTIFIER_VERBOSE
> > > opal_atomic_increment(counter);
> > > if (counter > threshold) {
> > > orte_notifier.api(.....)
> > > }
> > > #endif
> > >
> > > You would set the specific thresholds for each situation via MCA
> > params, so this could be tuned to fit specific needs.
> > > Those who don't want the penalty can just build normally - those
> > who want this level of information can enable it.
> > >
> > > We can then see just how much penalty is involved in real world
> > situations. My guess is that it won't be that big, but it's
> > > hard to know without seeing how frequently we actually insert this
> > code.
> > >
> > > Hope that makes sense
> > > Ralph
> > >
> > >
> > > On Wed, May 27, 2009 at 1:25 AM, Sylvain Jeaugey <sylvain.jeaugey_at_[hidden]
> > > wrote:
> > > About performance, I may miss something, but our first goal
> > was to track already slow pathes.
> > >
> > > We imagined that it could be possible to add at the
> > beginning (or end) of this "bad path" just one line that
> > > would basically do an atomic inc. So, in terms of CPU
> > cycles, something like 1 for the inc and maybe 1 jump
> > > before. Are a couple of cycles really an issue in slow
> > pathes (which take at least hundreds of cycles), or do
> > > you fear out-of-cache memory accesses - or something else ?
> > >
> > > As for outputs, they indeed are slow (and can slow down
> > considerably an application if not synchronized), but
> > > aggregation on the head node should solve our problems. And
> > if not, we can also disable outputs at runtime.
> > >
> > > So, in my opinion, no application should notice a difference
> > (unless you tune the framework to output every
> > > warning).
> > >
> > > Sylvain
> > >
> > >
> > > On Tue, 26 May 2009, Jeff Squyres wrote:
> > >
> > > Nadia --
> > >
> > > Sorry I didn't get to jump in on the other thread earlier.
> > >
> > > We have made considerable changes to the notifier framework
> > in a branch to better support "SOS"
> > > functionality:
> > >
> > > https://www.open-mpi.org/hg/auth/hgwebdir.cgi/jsquyres/opal-sos
> > >
> > > Cisco and Indiana U. have been working on this branch for a
> > while. A description of the SOS stuff is
> > > here:
> > >
> > > https://svn.open-mpi.org/trac/ompi/wiki/ErrorMessages
> > >
> > > As for setting up an external web server with hg, don't
> > bother -- just get an account at bitbucket.org.
> > > They're free and allow you to host hg repositories there.
> > I've used bitbucket to collaborate on code
> > > before it hits OMPI's SVN trunk with both internal and
> > external OMPI developers.
> > >
> > > We can certainly move the opal-sos repo to bitbucket (or
> > branch again off opal-sos to bitbucket --
> > > whatever makes more sense) to facilitate collaborating with
> > you.
> > >
> > > Back on topic...
> > >
> > > I'd actually suggest a combination of what has been
> > discussed in the other thread. The notifier can be
> > > the mechanism that actually sends the output message, but it
> > doesn't have to be the mechanism that tracks
> > > the stats and decides when to output a message. That can be
> > separate logic, and therefore be more
> > > fine-grained (and potentially even specific to the MPI
> layer).
> > >
> > > The Big Question will how to do this with zero performance
> > impact when it is not being used. This has
> > > always been the difficult issue when trying to implement any
> > kind of monitoring inside the core OMPI
> > > performance-sensitive paths. Even adding individual
> > branches has met with resistance (in
> > > performance-critical code paths)...
> > >
> > >
> > >
> > > On May 26, 2009, at 10:59 AM, Nadia Derbey wrote:
> > >
> > > Hi,
> > >
> > > While having a look at the notifier framework under
> > orte, I noticed that
> > > the way it is written, the init routine for the
> > selected module cannot
> > > be called.
> > >
> > > Attached is a small patch that fixes this issue.
> > >
> > > Regards,
> > > Nadia
> > >
> > > <orte_notifier_fix_select.patch><ATT14046023.txt>
> > >
> > >
> > >
> > > --
> > > Jeff Squyres
> > > Cisco Systems
> > >
> > > _______________________________________________
> > > devel mailing list
> > > devel_at_[hidden]
> > > http://www.open-mpi.org/mailman/listinfo.cgi/devel
> > >
> > > _______________________________________________
> > > devel mailing list
> > > devel_at_[hidden]
> > > http://www.open-mpi.org/mailman/listinfo.cgi/devel
> > >
> > >
> > >
> > >
> >
> > <ATT14397892.txt>
>
>
> --
> Jeff Squyres
> Cisco Systems
>
> _______________________________________________
> devel mailing list
> devel_at_[hidden]
> http://www.open-mpi.org/mailman/listinfo.cgi/devel
>

-- 
Jeff Squyres
Cisco Systems