Re: regtest not catched by make check

[Thomas Morley]

Am So., 8. Nov. 2020 um 11:30 Uhr schrieb Jonas Hahnfeld <hahnjo@hahnjo.de>:
>
> Am Sonntag, den 08.11.2020, 11:20 +0100 schrieb Thomas Morley:
> > Am So., 8. Nov. 2020 um 09:28 Uhr schrieb Jonas Hahnfeld <hahnjo@hahnjo.de>:
> > >
> > > Am Sonntag, den 08.11.2020, 01:49 +0100 schrieb Thomas Morley:
> > > > To be honest, before, I had not thought very deeply about how
> > > > comparing regression tests with 'make  check' works.
> > > >
> > > > But now I think a regression test should actually _test_ something.
> > > > The test may succeed or fail.
> > > > If it fails, (but the snippet itself compiles), then the result should
> > > > be catched by 'make check'. In cases where this can't be made
> > > > possible, the snippet should error, stop 'make check' and a meaningful
> > > > error message should be returned.
> > >
> > > While this would be ideal, it might not always be possible. That's
> > > exactly why visual inspection may be required.
> >
> > As mentioned here
> > https://gitlab.com/lilypond/lilypond/-/merge_requests/497#note_442096848
> > I actually did visual inspection manually.
> > It was out of interest, not because I thought it was needed. Good I
> > did so otherwise I wouldn't have noticed my first attempt to fix the
> > issue was insufficient.
>
> IMHO manual testing of the feature should always be part of the
> development process.

Sure, I always have some local files testing the feature in question.
Additionally I use 'make check' to catch (corner-) cases which I may
have overlooked.

> > But there are not so many regtests for fret-diagrams, so it was not much 
> > work.
> > Of course I looked in the obvious regtests, what if the patch caused
> > some bad elsewhere?
> > There should be some script widely comparing things. Well, like 'make 
> > check'.
> > Other automated visual inspection is more a safety net. Imho, it
> > should not be part of the usual development process of single patches.
>
> I mentioned scripts/auxiliar/make-regtest-pngs.sh earlier in this
> thread, maybe this already does what you mean? If not, I may not
> understand what you're looking for...
>
> Jonas

Well, then we should make running make-regtest-pngs.sh a requirement
additionally to 'make check'.
I'll test how long it lasts on my weak laptop (I never did it before).

Cheers,
  Harm