Open MPI logo

Hardware Locality Development Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Hardware Locality Development mailing list

Subject: Re: [hwloc-devel] git / nightly builds
From: Jeff Squyres (jsquyres) (jsquyres_at_[hidden])
Date: 2013-09-29 08:00:54


On Sep 28, 2013, at 8:12 AM, Brice Goglin <Brice.Goglin_at_[hidden]> wrote:

>> Let's go through the use cases of what we want:
>>
>> 1. "make dist" in a developer's git clone. This should make a "hwloc-<git describe>.tar.*.
>
> This is actually the critical point, see below.
>
>> 2. make a nightly snapshot tarball. The more I think about this, the more I think it's the same as #1, right?
>
> Yes, that's why I didn't understand why the create_tarball script also
> modified VERSION to add git describe. These changes should be either
> entirely outside of make dist (if we want git describe for nightly but
> not for make dist) or entirely inside make dist (if we want for both).

Agreed. Let's have distscript.csh be the one that sets the git-describe value in VERSION (more below).

>> 3. make a release tarball, "hwloc-major.minor.releasegreek.tar.*".
>>
>> Are these the three (or two, if #2 is the same as #1) use cases we want? If so, I can see about making the code simpler -- e.g., I didn't know about overriding the VERSION Makefile macro trick...
>
> Changing VERSION on the command-line doesn't change the lstopo
> --version, so it may not be what we really want. Also, if changing the
> suffix is just a sed on VERSION file before autogen or configure, it's
> fine too.
>
> This all depends on what we really want for (1).
> * If we don't do (1), we can remove tons of lines of code from the
> configury and just have the nightly and release scripts modify VERSION
> before running autogen. Easy.
> * If we do (1), that needs much more work.

I'm not sure I agree. Let me review the 3 scripts and what each does (and why):

1. config/distscript.csh:
   - Called by "make dist[check]"
   - Does 3 main things:
     a) Sets git-describe value in VERSION file (but only if snapshot=1)
     b) Generates/copies doxygen output files to the dist tree
        --> We wanted to not require users to have doxygen
        --> We also wanted to ship final doxygen output in tarballs
        --> This created quite a bit of complexity in itself
     c) Get new config.guess and config.sub files from git.savannah.gnu.org
        --> This code is from when GNU Autotools releases were few and far between; it may not be necessary any more. We might well be able to just remove this code and still be fine.

2. contrib/nightly/make_snapshot_tarball:
   - Invoked via cron on the build machine
   - Very specifically written to interact with the web download tree
   - Generally does the following:
     a) Gets a new git clone (into a unique directory)
     b) Compares output from "git describe ..." to latest_snapshot.txt
     c) If they're the same, exit 0
        --> If there are no commits since last tarball, no need to do anything
     d) Run autogen, configure, make distcheck.
     e) Move resulting tarballs to the web download directory
     e) Re-generate md5sums.txt/sha1sums.txt

3. contrib/dist/make_dist_tarball:
   - Invoked manually by a human when making releases
   - Generally does the following:
     a) Check versions of GNU Autotools
        --> To ensure to use *the same* versions of the Autotools for an entire release series
     b) Update VERSION file with the release date
     c) Run autogen + configure, build doxygen docs, build README, run make distcheck
     d) Remove "greek" value from VERSION and run c) again

> I actually don't care much about (1), I am used to tarballs without the
> SVN revision suffix (not sure why I didn't always get that suffix). I
> agree that it's convenient to have the suffix for developer builds (when
> you want to compare several of them, when you distribute that tarball
> for some reason, etc). But maybe the nightly script is enough for these
> cases? Does it work with locally modified trees? Or trees that contain
> additional commits?

No, it gets a fresh clone every night. The intent was that this script runs in an automated fashion, and sometimes the build fails. So it leaves the failed clone/build in a place where a human can go post-mortem the tree and figure out why the build failed.

But to be fair, this hasn't been a big issue for hwloc (it has with Open MPI -- e.g., "make dist" works for developers, but it failed on the build machine. So it was really helpful to be able to login as mpiteam_at_build_machine and go look into the build tree and see WTF happened).

It also specifically interacts with the web download tree. To be clear: this script is all about deciding whether to make a new snapshot, and if so, run "make dist" to do so, and then copies to results to the web tree.

The core of this is:

- *all* use cases run "make dist"; config/distscript.csh is used by all of them
- make_dist_tarball is some additional accounting surrounding "make dist"
- make_snapshot_tarball is some additional (different) accounting surrounding "make dist"

All that being said, I think 2 immediate improvements/simplifications are:

- have make_snapshot_tarball not set "git describe" in VERSION
- remove the config.guess/config.sub fetching from distscript.csh

I'll commit those now.

However, for the other cases, I think that doxygen is our main culprit for complexity. :-\ Meaning: I'm now not sure how to make them simpler...

Thoughts?

> By the way, maybe move that script back from nightly to dist.

Sure; I don't have much of an opinion here.

>> Do you have a favorite git commit emailing script? We can probably use the generic github "WebHook URLs" hook (at the top of the list) and host the diff-emailing script at IU (or anywhere, actually).
>
> I use the attached one for Open-MX and KNEM. Not sure I tried many of
> them, but this one worked fine so far. It generates messages such as:
>
> http://lists.gforge.inria.fr/pipermail/knem-commits/2013-July/000465.html
> I don't think it truncates the diff yet. We may want some separators
> between commits too. All this shouldn't be hard to configure.

It looks like github sends an HTTP post with the following information in it:

    https://help.github.com/articles/post-receive-hooks

Do you have an easy place to host your script to try it out? Or do you want to wait for IU to host it (i.e., tomorrow)?

-- 
Jeff Squyres
jsquyres_at_[hidden]
For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/