Re: Automatically checking regtests (was: Re: Minor releases?)

[Trevor Daniels]

Graham Percival wrote Monday, August 17, 2009 10:35 PM


> On Mon, Aug 17, 2009 at 05:17:08PM +0200, Reinhold Kainhofer 
> wrote:
> > Am Montag, 17. August 2009 16:08:36 schrieb Michael Käppler:
> > >
> > > > (nobody checks the regression tests for each release, for 
> > > > example
> > > > -- and that's trivially done with a web browser!)
> > >
> > > That reminds me of an idea I recently had: Wouldn't it be 
> > > possible to
> > > automatically generate a sort of "checksum" for each 
> > > regression-test
> > > output-file and compare it with the former releases?
> >
> > Isn't this exactly what we already have (make test-baseline to 
> > create the
> > reference version and then "make test" to compare the new output 
> > to the
> > reference output...). For the results see
> >       http://lilypond.org/test/
> > The 2.13.3 results are at:
> >
> > http://lilypond.org/test/v2.13.3-0/compare-v2-13/index.html)
>
> Yes.  Right now there happen to be a lot, but I think there's
> generally around a dozen examples.

They're almost all due to the different
beaming rules, and these are all changed
for the better, IMHO.

> > Graham was referring to the fact that nobody seem to bother about 
> > looking at
> > those automatically-created regression results before or after a 
> > release.
>
> Yes.  All it takes is bookmarking the site, checking it whenever
> there's a release, and reporting any broken examples.  However,
> nobody is willing to commit to do this.  15 minutes whenever
> there's a release, which happens at most once every two weeks.

I'd not looked at this before.  It's very
impressive!  Since I usually download and
install each release I'll commit to looking
at it before I download and shouting if there
is a serious difference.  What in general is
the base release used in the comparison?  Is
it the previous minor release or the previous
stable release?

Trevor


Trevor