Re: regtest not catched by make check

[Thomas Morley]

Am So., 8. Nov. 2020 um 09:28 Uhr schrieb Jonas Hahnfeld <hahnjo@hahnjo.de>:
>
> Am Sonntag, den 08.11.2020, 01:49 +0100 schrieb Thomas Morley:
> > To be honest, before, I had not thought very deeply about how
> > comparing regression tests with 'make  check' works.
> >
> > But now I think a regression test should actually _test_ something.
> > The test may succeed or fail.
> > If it fails, (but the snippet itself compiles), then the result should
> > be catched by 'make check'. In cases where this can't be made
> > possible, the snippet should error, stop 'make check' and a meaningful
> > error message should be returned.
>
> While this would be ideal, it might not always be possible. That's
> exactly why visual inspection may be required.

As mentioned here
https://gitlab.com/lilypond/lilypond/-/merge_requests/497#note_442096848
I actually did visual inspection manually.
It was out of interest, not because I thought it was needed. Good I
did so otherwise I wouldn't have noticed my first attempt to fix the
issue was insufficient.
But there are not so many regtests for fret-diagrams, so it was not much work.
Of course I looked in the obvious regtests, what if the patch caused
some bad elsewhere?
There should be some script widely comparing things. Well, like 'make check'.
Other automated visual inspection is more a safety net. Imho, it
should not be part of the usual development process of single patches.

> > In the current case I finally found something catchable, basically I
> > print a Staff with separated markups (_not_ a single markup). The
> > entire markups are sized differently if the test succeeds or fails.
>
> This sounds like the regtest is getting even more complex than the
> initial code you shared in this thread. You should weight if this is
> worth the benefit it provides; test code should usually so trivial that
> it doesn't need testing itself.
>
> FWIW fiddling around with custom stencils just to get something 'make
> check' can recognize is the wrong trade-off IMO.
>
> Jonas
>
> > I'll upload a new patch tomorrow, very late here.
> >
> > Thanks,
> >   Harm
>

The snippet I found is not more complex than the one initially posted
in this thread, rather it arranges things differently.
I think I'll upload a separate patch for this. Then we can continue
discussion during review.


Thanks,
  Harm