Open MPI logo

Hardware Locality Development Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Hardware Locality Development mailing list

Subject: Re: [hwloc-devel] git / nightly builds
From: Jeff Squyres (jsquyres) (jsquyres_at_[hidden])
Date: 2013-09-29 08:00:54

On Sep 28, 2013, at 8:12 AM, Brice Goglin <Brice.Goglin_at_[hidden]> wrote:

>> Let's go through the use cases of what we want:
>> 1. "make dist" in a developer's git clone. This should make a "hwloc-<git describe>.tar.*.
> This is actually the critical point, see below.
>> 2. make a nightly snapshot tarball. The more I think about this, the more I think it's the same as #1, right?
> Yes, that's why I didn't understand why the create_tarball script also
> modified VERSION to add git describe. These changes should be either
> entirely outside of make dist (if we want git describe for nightly but
> not for make dist) or entirely inside make dist (if we want for both).

Agreed. Let's have distscript.csh be the one that sets the git-describe value in VERSION (more below).

>> 3. make a release tarball, "hwloc-major.minor.releasegreek.tar.*".
>> Are these the three (or two, if #2 is the same as #1) use cases we want? If so, I can see about making the code simpler -- e.g., I didn't know about overriding the VERSION Makefile macro trick...
> Changing VERSION on the command-line doesn't change the lstopo
> --version, so it may not be what we really want. Also, if changing the
> suffix is just a sed on VERSION file before autogen or configure, it's
> fine too.
> This all depends on what we really want for (1).
> * If we don't do (1), we can remove tons of lines of code from the
> configury and just have the nightly and release scripts modify VERSION
> before running autogen. Easy.
> * If we do (1), that needs much more work.

I'm not sure I agree. Let me review the 3 scripts and what each does (and why):

1. config/distscript.csh:
   - Called by "make dist[check]"
   - Does 3 main things:
     a) Sets git-describe value in VERSION file (but only if snapshot=1)
     b) Generates/copies doxygen output files to the dist tree
        --> We wanted to not require users to have doxygen
        --> We also wanted to ship final doxygen output in tarballs
        --> This created quite a bit of complexity in itself
     c) Get new config.guess and config.sub files from
        --> This code is from when GNU Autotools releases were few and far between; it may not be necessary any more. We might well be able to just remove this code and still be fine.

2. contrib/nightly/make_snapshot_tarball:
   - Invoked via cron on the build machine
   - Very specifically written to interact with the web download tree
   - Generally does the following:
     a) Gets a new git clone (into a unique directory)
     b) Compares output from "git describe ..." to latest_snapshot.txt
     c) If they're the same, exit 0
        --> If there are no commits since last tarball, no need to do anything
     d) Run autogen, configure, make distcheck.
     e) Move resulting tarballs to the web download directory
     e) Re-generate md5sums.txt/sha1sums.txt

3. contrib/dist/make_dist_tarball:
   - Invoked manually by a human when making releases
   - Generally does the following:
     a) Check versions of GNU Autotools
        --> To ensure to use *the same* versions of the Autotools for an entire release series
     b) Update VERSION file with the release date
     c) Run autogen + configure, build doxygen docs, build README, run make distcheck
     d) Remove "greek" value from VERSION and run c) again

> I actually don't care much about (1), I am used to tarballs without the
> SVN revision suffix (not sure why I didn't always get that suffix). I
> agree that it's convenient to have the suffix for developer builds (when
> you want to compare several of them, when you distribute that tarball
> for some reason, etc). But maybe the nightly script is enough for these
> cases? Does it work with locally modified trees? Or trees that contain
> additional commits?

No, it gets a fresh clone every night. The intent was that this script runs in an automated fashion, and sometimes the build fails. So it leaves the failed clone/build in a place where a human can go post-mortem the tree and figure out why the build failed.

But to be fair, this hasn't been a big issue for hwloc (it has with Open MPI -- e.g., "make dist" works for developers, but it failed on the build machine. So it was really helpful to be able to login as mpiteam_at_build_machine and go look into the build tree and see WTF happened).

It also specifically interacts with the web download tree. To be clear: this script is all about deciding whether to make a new snapshot, and if so, run "make dist" to do so, and then copies to results to the web tree.

The core of this is:

- *all* use cases run "make dist"; config/distscript.csh is used by all of them
- make_dist_tarball is some additional accounting surrounding "make dist"
- make_snapshot_tarball is some additional (different) accounting surrounding "make dist"

All that being said, I think 2 immediate improvements/simplifications are:

- have make_snapshot_tarball not set "git describe" in VERSION
- remove the config.guess/config.sub fetching from distscript.csh

I'll commit those now.

However, for the other cases, I think that doxygen is our main culprit for complexity. :-\ Meaning: I'm now not sure how to make them simpler...


> By the way, maybe move that script back from nightly to dist.

Sure; I don't have much of an opinion here.

>> Do you have a favorite git commit emailing script? We can probably use the generic github "WebHook URLs" hook (at the top of the list) and host the diff-emailing script at IU (or anywhere, actually).
> I use the attached one for Open-MX and KNEM. Not sure I tried many of
> them, but this one worked fine so far. It generates messages such as:
> I don't think it truncates the diff yet. We may want some separators
> between commits too. All this shouldn't be hard to configure.

It looks like github sends an HTTP post with the following information in it:

Do you have an easy place to host your script to try it out? Or do you want to wait for IU to host it (i.e., tomorrow)?

Jeff Squyres
For corporate legal information go to: