Open MPI logo

Hardware Locality Development Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Hardware Locality Development mailing list

Subject: Re: [hwloc-devel] git / nightly builds
From: Brice Goglin (Brice.Goglin_at_[hidden])
Date: 2013-09-28 04:41:58


Jeff,

I am having lots of problems when switching the regression testing stuff
(jenkins) to git, mostly because of make dist. Actually it worked 2 days
ago, no it breaks because hwloc-unknown.tar.* remain after make distcheck.

1) There's something I don't understand in the dist scripts.
config/distscript.csh is only called the top-level Makefile.am, with 4th
argument = HWLOC_SVN_R, which is never set. So we always fallback to
git-describe. When building from a tarball, you get "unknown". That
seems to break make distcheck. We need to pass something in that
argument, but I don't see what.

2) VPATH dist support is more fragile than before (I always build under
$srcdir/build). In the past, we could do a VPATH make dist by just
symlinking srcdir/doc/doxygen-doc to builddir/doc/doxygen-doc. This now
generates "unknown" tarballs because we check for .git existence
explicitly. I fixed my own case by checking for ../.git as well but
that's not satisfying.

It looks like we can fix this by checking for $srcdir/.git instead. If
we want VPATH support where $builddir isn't a child of $srcdir, we'll
have to set GIT_DIR=$srcdir/.git before calling git-describe too.

I think this is becoming way too complicated. Nobody won't be able to
maintain that code in a couple years. Worse, what if you leave Cisco and
stop working on hwloc one day? In my other projects, I would just
override the VERSION makefile variable on the make command-line to
change the tarball name (you won't get that VERSION in lstopo --version,
but you would still see 1.8a1 from configure). We should rethink what we
actually need here. For instance if we can simpify things by not
building 1.8-final when we build 1.8-rcX, that'd be fine with me.

Random other questions:
* where do you configure commit emails? can we drop/change the
open-mpi/hwloc subject prefix? I removed the hwloc-svn prefix from
mailman to avoid double-prefixing for now
* can we get commit diffs in email, with some truncation limit to 50kB
or so?

Le 27/09/2013 15:36, Jeff Squyres (jsquyres) a écrit :
> I'd say that git and the new Trac are now fully open for business. I might still do some shifting around of tags (see below), but otherwise, I think everything is just about ready.
>
> I'm revamping make_dist_script and the nightly build script; I should finally be able to commit those today, if all goes well. However, I had to make some changes to VERSION and some other surrounding infrastructure to make it all work. Specifically: git just does some things *differently* than SVN, so it required some changes in the way that hwloc does things, infrastructure-wise.
>
> Two things:
>
> 1. I'm not excited about back-porting these changes all the way back to hwloc-1.0 just so that we can make nightly tarballs for these branches which aren't used any more. I'm thinking that I should definitely make these changes for master and the v1.7 branch. Should I also do the v1.6 branch?
>
> I'm thinking that back-porting further is useless; we should just stop making nightlies for all the older branches.
>
> To clarify: with SVN, we still checked all the older branches every night and would make a new tarball if there was ever a change. We never made changes in those older branches, so we never made new nightlies. But we *would have*, if a change had every been committed on those branches.
>
> 2. With SVN, adding the r number in the tarball version was sufficient to observe ordering of the tarballs. For example:
>
> hwloc-1.8r1234.tar.bz2
> hwloc-1.8r1240.tar.bz2
> hwloc-1.8r1255.tar.bz2
> ...etc.
>
> With git, we only have a hash number. So there's no obvious ordering of the tarballs. For example:
>
> hwloc-1.8git11223344.tar.bz2
> hwloc-1.8git00aa3344.tar.bz2
> hwloc-1.8git99aa2277.tar.bz2
> ...etc.
>
> However, there is "git describe" functionality which, in our case, can show you how many commits you are beyond a given tag. For example, I added a "dev" tag on the master branch for the first pure-git commit (i.e., 1 beyond the last SVN commit). For example:
>
> $ git clone git_at_[hidden]:open-mpi/hwloc.git
> ...clone completes successfully...
> $ cd hwloc
> $ git checkout master
> $ git describe --always
> dev-3-g51efdd1
>
> Where the "3" represents that we're 3 commits beyond the "dev" tag. The "g" just means "git", and the "51efdd1" is the hash of that commit.
>
> Hence, if we use that as our version string, we get tarballs named something like this:
>
> hwloc-dev-3-g51efdd1.tar.bz2
> hwloc-dev-10-gf50a385.tar.bz2
> ...etc.
>
> For the master branch, I think that's fine. However, note that ***THIS IS DIFFERENT THAN WHAT WE HAVE PREVIOUSLY DONE ON RELEASE BRANCHES!***
>
> For example, if you checkout the v1.7 branch:
>
> $ git checkout v1.7
> $ git describe --always
> hwloc-1.7.2-4-g3a6f84c
>
> *** NOTE THE DIFFERENCE HERE:
>
> a) The last SVN nightly snapshot on the v1.7 branch was named
> hwloc-1.7.3rc1r5779.tar.bz2.
> b) The first git nightly snapshot on the v1.7 branch will be named
> hwloc-1.7.2-4-g3a6f84c.tar.bz2.
>
> Note "1.7.3rc1..." vs. "1.7.2...". I.e., the git name will say "we're X commits beyond the 1.7.2 tag", but the old SVN name was "we're at this *upcoming* version".
>
> I think that this is ok (other projects use this "git describe" kind of behavior for their nightly snapshots), but this is a change from what we used to do, so I wanted to call it out specifically.
>
> Are you guys ok with this?
>
> (note: I'm still mucking with the final tag names, so some of the above may change slightly)
>