Re: Integration of git-based release workflow into "make dist"

[Roger Leigh]

On Fri, Aug 14, 2009 at 07:53:46PM +0200, Ralf Wildenhues wrote:
> Hello Roger,
> 
> * Roger Leigh wrote on Fri, Aug 14, 2009 at 11:40:18AM CEST:
> > 
> > An initial implementation follows.  This works, but it does need
> > further refinement (error checking, for example).  And probably
> > review by a git expert.  I'm sure other people can make it much
> > nicer, but this hopefully demonstrates the point.
> 
> Thanks.  Not to sound discouraging, but this seems to be able to cope
> mostly without any Automake internals (am__remove_distdir is a rather
> benign issue) nor need special help from Automake.  That means it can
> easily live outside of Automake proper for a while at least, until it
> is settled and useful for more than a couple of projects or distros,
> as Bob already noted.  You can keep it synchronized across projects
> easily by putting the code into a fragment.am file and including that
> into your toplevel Makefile.am, and synchronizing the fragment, no?

Absolutely.  I am perfectly happy in doing so.

I've attached a new version to this mail, with these changes:

• error checking
• VPATH support
• checking for already-tagged repos
• tagging of both release and distribution branches
• customisation of both release and distribution tagging behaviour,
  as well as allowing it to be skipped e.g.
  make dist-git GIT_RELEASE_TAG=false

> You could consider submitting this fragment as a gnulib module (where
> you just place your code into the Makefile.am part of the modules/..
> file.  Or as part of maint.mk, if the gnulib crowd likes that better.

I hadn't come across this before; I didn't realise it had Makefile.am
fragments as well.  That's perfectly OK by me.

How does one extract these bits into a project's Makefile.am
automatically?  Or is it just a matter of copying and pasting the
needed bits?

* Roger Leigh wrote on Sat, Aug 15, 2009 at 12:13:21PM CEST:
> > Additionally, VCSes don't typically store the results of "make dist",
> > only the source for that.  Due to changing tool versions over the
> > years, you can get in a situation where you can no longer rebootstrap
> > checkouts of old versions.  By storing this in the VCS as well, you
> > guarantee that you can get a usable tree even if those tools are no
> > longer available or functional.  Plus, you can also diff the
> > generated files between releases as an added bonus, which can help
> > track down build infrastructure issues.
> 
> I think your solution is pretty cool, and goes to show again how nice it
> is that git is so versatile.  I might consider using it for future
> Automake releases.
> 
> But as to inclusion in Automake proper, I'm still slightly on the side
> of "this is git-centric and probably a bit Debian-centric" in the sense
> that other distros or other developers might want to choose to do it a
> bit differently, and I'd rather have Automake offer generic solutions.
> Are you coordinating with other distros?  Some feedback from others
> would certainly not hurt here.

I'm going to run it past the git list in case there are any technical
flaws in what I'm doing.

I haven't yet discussed it with anyone else; I thought bringing it up
on this list would be best initially.  I think other git-using
automake users are the best people to provide feedback, rather than
distributors, since they are the primary audience.  I'm not quite
sure how best to contact them though, other than this list and the
git list.

I've made the release and distribution tag and commit stuff completely
configurable by make variables.  However, you are correct that this is
rather git-specific.  There are problems making this more general:

• different VCSes have different restrictions on tag names; CVS doesn't
  allow '.' so will break on $(VERSION), others don't allow '/'.  The
  default tag name scheme will out of necessity need to be tailored to
  each VCS.  Different projects also have different rules, so it's
  highly likely this will need to be customised from whatever default
  we pick.

• Not all VCSes support GPG signing of commits and/or tags.  This
  is geared to signing tags, but could be generalised to also allow
  signing commits.

• VCS branch naming limitations; the branch name chosen here contains
  no odd characters, so should be fine.  (Unless you're using GNU Arch.
  But then you have bigger problems!)

The design of the rest (rather than this specific implementation) is
generic: committing a copy of the distdir onto a specified branch
is possible in all VCSes I know; the difficultly of actually
implementing this in automake is the main challenge here.

It might be worth bringing up this on the relevent VCS mailing lists
in order to generalise what can be sensibly generalised.

However, I don't think it should be a requirement to support multiple
version control systems /at the same time/.  One typically only
develops in a particular VCS type, even if it gets mirrored in others.

In your earlier mail:
> As to package versioning, see this discussion and references therein:
> <http://thread.gmane.org/gmane.comp.sysutils.automake.general/10720>
> as well as the approach currently used in gnulib's GNUmakefile.
> (I still haven't gotten very far with a nonworking patch on this.)

I thought that use of $(PACKAGE) and $(VERSION) were still supported
for the time being?

I read this thread with interest at the time, and adopted a variant
of this into my workflow.  I'll detail that in a separate mail for
anyone who is interested.

With regard to getting at PACKAGE and VERSION, it would be great if
autoconf/automake could standardise on some hook mechanism for
querying the source package for this information.  I'll detail what
I did in a separate mail, since it's not specifically related to the
dist-git functionality.


Regards,
Roger


This is the current code:

GIT_RELEASE_BRANCH=HEAD
GIT_RELEASE_TAG=true
GIT_RELEASE_TAG_SIGN=true
GIT_RELEASE_TAG_NAME=release/$(PACKAGE)-$(VERSION)
GIT_RELEASE_TAG_MESSAGE="Release of $(PACKAGE)-$(VERSION)"

GIT_DIST_BRANCH=distrib
GIT_DIST_COMMIT_MESSAGE="Distribution of $(PACKAGE) version $(VERSION)"
GIT_DIST_TAG=true
GIT_DIST_TAG_SIGN=true
GIT_DIST_TAG_NAME=distribution/$(PACKAGE)-$(VERSION)
GIT_DIST_TAG_MESSAGE="Distribution of $(PACKAGE)-$(VERSION)"

dist-git: distdir
        cd "$(abs_top_srcdir)"; \
        if [ ! -d .git ]; then \
            echo "dist-git: Not a git repository" 1>&2; \
            exit 1; \
        fi; \
        if [ "$(GIT_RELEASE_TAG)" = "true" ]; then \
          if git show-ref --tags -q $(GIT_RELEASE_TAG_NAME); then \
            echo "git release tag $(GIT_RELEASE_TAG_NAME) already exists; not 
distributing" 1>&2; \
            exit 1; \
          fi; \
        fi; \
        if [ "$(GIT_DIST_TAG)" = "true" ]; then \
          if git show-ref --tags -q $(GIT_DIST_TAG_NAME); then \
            echo "git distribution tag $(GIT_DIST_TAG_NAME) already exists; not 
distributing" 1>&2; \
            exit 1; \
          fi; \
        fi; \
        DISTDIR_INDEX="$(abs_top_builddir)/$(distdir).git.idx"; \
        DISTDIR_TREE="$(abs_top_builddir)/$(distdir)"; \
        rm -f "$$DISTDIR_INDEX"; \
        GIT_INDEX_FILE="$$DISTDIR_INDEX" GIT_WORK_TREE="$$DISTDIR_TREE" git add 
-A || exit 1; \
        GIT_INDEX_FILE="$$DISTDIR_INDEX" TREE="$$(git write-tree)"; \
        rm -f "$$DISTDIR_INDEX"; \
        [ -n "$$TREE" ] || exit 1; \
        RELEASE_HEAD="$$(git show-ref -s $(GIT_RELEASE_BRANCH))"; \
        COMMIT_OPTS="-p $$RELEASE_HEAD"; \
        DIST_PARENT="$$(git show-ref --heads -s 
refs/heads/$(GIT_DIST_BRANCH))"; \
        if [ -n "$$DIST_PARENT" ]; then \
          COMMIT_OPTS="$$COMMIT_OPTS -p $$DIST_PARENT"; \
        fi; \
        COMMIT="$$(echo $(GIT_DIST_COMMIT_MESSAGE) | git commit-tree "$$TREE" 
$$COMMIT_OPTS)"; \
        [ -n "$$COMMIT" ] || exit 1; \
        git update-ref "refs/heads/$(GIT_DIST_BRANCH)" "$$COMMIT" 
"$$DIST_PARENT" || exit 1;\
        if [ "$(GIT_RELEASE_TAG)" = "true" ]; then \
          RELEASE_TAG_OPTS=""; \
          if [ "$(GIT_RELEASE_TAG_SIGN)" = "true" ]; then \
            RELEASE_TAG_OPTS="$$TAG_OPTS -s"; \
          fi; \
          git tag -m $(GIT_RELEASE_TAG_MESSAGE) $$RELEASE_TAG_OPTS 
"$(GIT_RELEASE_TAG_NAME)" "$$COMMIT" || exit 1; \
            echo "release tagged as $(GIT_RELEASE_TAG_NAME)"; \
        fi; \
        if [ "$(GIT_DIST_TAG)" = "true" ]; then \
          DIST_TAG_OPTS=""; \
          if [ "$(GIT_DIST_TAG_SIGN)" = "true" ]; then \
            DIST_TAG_OPTS="$$TAG_OPTS -s"; \
          fi; \
          git tag -m $(GIT_DIST_TAG_MESSAGE) $$DIST_TAG_OPTS 
"$(GIT_DIST_TAG_NAME)" "$$COMMIT" || exit 1; \
            echo "distribution tagged as $(GIT_DIST_TAG_NAME)"; \
        fi;
        $(am__remove_distdir)

-- 
  .''`.  Roger Leigh
 : :' :  Debian GNU/Linux             http://people.debian.org/~rleigh/
 `. `'   Printing on GNU/Linux?       http://gutenprint.sourceforge.net/
   `-    GPG Public Key: 0x25BFB848   Please GPG sign your mail.

signature.asc
Description: Digital signature