Open MPI logo

Hardware Locality Development Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Hardware Locality Development mailing list

Subject: Re: [hwloc-devel] git / nightly builds
From: Paul Hargrove (phhargrove_at_[hidden])
Date: 2013-09-28 16:47:05


I am watching with intense interest, as GASNet will be moving to git on Nov
Could you give me a pointer to where I can get copies all the scripts that
you guys use for nightly tarballs, commit emails and such?
I'll want to have look after you have all the wrinkles ironed out to your

FWIW: a viewpoint from somebody who has yet to actually try to implement
his ideas:

We have PLANNED to script our nightlies to be named with a date stamp and 6
or 8 chars of the SHA1.
The format would be something like PROJECT-BRANCH-20130928-12abcdef.tar.gz
Where BRANCH=v<major>.<minor> for the UPCOMING release in most scenarios
(but could be a named feature branch like "oshmem")
That way directory listings would sort by branch and date (simple for mere
humans) while the SHA1 would allow fetching the corresponding version from
git. The full SHA1 would, of course, also be in a file in the tarball
(actual filename TBD).

I don't think we consider "git describe" to be a useful naming mechanism
for nightlies, though for other snapshots it might be useful.
For RCs, we pan to tag something like "1.7.3RC" at the start of the RC
cycle to get "git describe" to give names containing "1.7.3RC-<N>-<hash>"
which make some sense at a glance, even though N is incremented with every
commit and may grow much higher than a contentional RC number would.
 Again, "1.7.3" in this example is the UPCOMING release rather than the
previous one.

For a developer's "make dist" from a developer's clone, however, I think we
agree "git describe" is as good as anything else readily available.

So, in short: I think our plan aligns with yours on scenarios #1 ("make
dist" by Jane Developer) and #3 (official releases), but we wanted
something more people-friendly for #2 (nightly tarballs).


On Sat, Sep 28, 2013 at 4:30 AM, Jeff Squyres (jsquyres) <jsquyres_at_[hidden]
> wrote:

> On Sep 28, 2013, at 4:41 AM, Brice Goglin <Brice.Goglin_at_[hidden]> wrote:
> > I am having lots of problems when switching the regression testing stuff
> > (jenkins) to git, mostly because of make dist. Actually it worked 2 days
> > ago, no it breaks because hwloc-unknown.tar.* remain after make
> distcheck.
> This means the versions junk still isn't right yet. :-\
> > 1) There's something I don't understand in the dist scripts.
> > config/distscript.csh is only called the top-level, with 4th
> > argument = HWLOC_SVN_R, which is never set. So we always fallback to
> > git-describe. When building from a tarball, you get "unknown". That
> > seems to break make distcheck. We need to pass something in that
> > argument, but I don't see what.
> Good catch; sorry about that. HWLOC_SVN_R no longer exists (as you
> noted). I just removed that 4th argument to distscript.csh. Now,
> distscript (on master and v1.7) only edits VERSION if snapshot=1 and
> snapshot_version is empty (i.e., if you do "make dist" in a git clone
> instead of running contrib/nightly/make_nightly_snapshot, which will edit
> VERSION before running "make distcheck").
> > 2) VPATH dist support is more fragile than before (I always build under
> > $srcdir/build). In the past, we could do a VPATH make dist by just
> > symlinking srcdir/doc/doxygen-doc to builddir/doc/doxygen-doc. This now
> > generates "unknown" tarballs because we check for .git existence
> > explicitly. I fixed my own case by checking for ../.git as well but
> > that's not satisfying.
> Mmm. I preserved your ../.git check in
> .
> I don't think I knew/realized you were doign VPATH dist builds by doing
> the sym link.
> > It looks like we can fix this by checking for $srcdir/.git instead. If
> > we want VPATH support where $builddir isn't a child of $srcdir, we'll
> > have to set GIT_DIR=$srcdir/.git before calling git-describe too.
> >
> > I think this is becoming way too complicated. Nobody won't be able to
> > maintain that code in a couple years. Worse, what if you leave Cisco and
> > stop working on hwloc one day? In my other projects, I would just
> > override the VERSION makefile variable on the make command-line to
> > change the tarball name (you won't get that VERSION in lstopo --version,
> > but you would still see 1.8a1 from configure). We should rethink what we
> > actually need here.
> Yes, these are good points. I agree: the system is too complicated right
> now. :-\
> Let's go through the use cases of what we want:
> 1. "make dist" in a developer's git clone. This should make a "hwloc-<git
> describe>.tar.*.
> 2. make a nightly snapshot tarball. The more I think about this, the more
> I think it's the same as #1, right?
> 3. make a release tarball, "hwloc-major.minor.releasegreek.tar.*".
> Are these the three (or two, if #2 is the same as #1) use cases we want?
> If so, I can see about making the code simpler -- e.g., I didn't know
> about overriding the VERSION Makefile macro trick...
> > For instance if we can simpify things by not
> > building 1.8-final when we build 1.8-rcX, that'd be fine with me.
> I think that part is actually fairly simple; it just runs "make dist",
> strips out the greek value from VERSION, and runs "make dist" again.
> > Random other questions:
> > * where do you configure commit emails? can we drop/change the
> > open-mpi/hwloc subject prefix? I removed the hwloc-svn prefix from
> > mailman to avoid double-prefixing for now
> > * can we get commit diffs in email, with some truncation limit to 50kB
> > or so?
> Yeah, I'm a bit disappointed by the github email service hook (the config
> is here:; scroll down to
> "Email"). There's *very* little configuration available:
> - the address to send to
> - an email address secret
> - what address to send from
> There's no option for diffs (!), and no option to customize the
> mail/subject. :-\
> Do you have a favorite git commit emailing script? We can probably use
> the generic github "WebHook URLs" hook (at the top of the list) and host
> the diff-emailing script at IU (or anywhere, actually).
> --
> Jeff Squyres
> jsquyres_at_[hidden]
> For corporate legal information go to:
> _______________________________________________
> hwloc-devel mailing list
> hwloc-devel_at_[hidden]

Paul H. Hargrove                          PHHargrove_at_[hidden]
Future Technologies Group
Computer and Data Sciences Department     Tel: +1-510-495-2352
Lawrence Berkeley National Laboratory     Fax: +1-510-486-6900