Re: testing out Docker CI scripts?

[David Kastrup]

Han-Wen Nienhuys <address@hidden> writes:

> On Sat, Feb 22, 2020 at 5:23 PM Jonas Hahnfeld <address@hidden> wrote:
>> > I would be interested in your feedback.
>>
>> Not having run any of this, my immediate response would that it's not
>> running 'make doc' AFAICS.
>
> For changes to the code, it should be irrelevant to run make doc: the
> regression test should cover all the behaviors we care about from a
> programming perspective.

It would be nice if you considered asking questions instead of just
assuming that our established procedures do not make sense and are not
actually rooted in any relevant experience.

We had a considerable amount of documentation building failures due to
"code-only" changes that "couldn't possibly" affect the doc build.  To a
good degree this is because the in-code documentation (like of
properties, music functions and a whole bunch of stuff ending up in the
Internals Reference) is run through various interpreters of Texinfo.

As a consequence of experiencing this more often than one naïvely
thought possible, we brainstormed about how we could make our tests more
reliable.  Exactly because the average developer cannot be expected to
make a doc build for every little change they cannot imagine to affect
the ability of make doc to complete, we added the doc build to the
automated procedures expected to function as a safety net.

> The time that David quotes for 'make doc' (~40 minutes) sounds wrong.
>
> $ ls -1 input/regression/*.ly|wc
>    1347    1347   55843
>
> $ grep @lilypond $(find  Documentation/ -name '*.*tely' | grep -v
> 'Documentation/[a-z][a-z]/')|wc
>    1828    1938  136964
>
> Building the docs should take about 1.5x the time of building the
> regtests. lilypond-book uses a shared database for snippets across all
> languages, so there should be neglible additional cost for the rest of
> the languages.

Assuming that all translations are kept at exactly the same level and
the example code is not at all adapted to the language in question in
lyrics and code comments.

And assuming that our regtests make up 40% of the LilyPond material in
the documentation and are not getting recompiled as part of including
them into the various forms of the documentation.

Since the documentation graphics are produced in PNG format for HTML
inclusion and in PDF format for PDF inclusion, that seems audacious.

> If CI becomes faster and cheaper, it will be easier to have instant
> and automatic feedback on all versions of a patch.

A significant part of the feedback is the required runtime.  One would
have to make sure to get reasonable numbers for that when the actual
runtime is getting masked.

>> On a related note, I think we should likely decide on our future
>> directions with respect to tooling first. Once we settle on Gerrit or
>> GitLab (or something completely else) both environments have their
>> own possibilities of integrating CI systems. I've been busy last week
>> and will be until next weekend, or I would have started a thread to
>> move this forwards.
>
> The CI "system" (jenkins, gitlab pipelines etc.) is somewhat
> orthogonal from the container setup (defining docker images etc.), so
> I think there is not so much double work. The scripts I'm offering
> here also have the advantage that you can run them off git commits
> from the local development environment.
>
> I'm curious what you come up with for CI tooling. If a full run takes
> 122 CPU minutes, we'll fall out of the free tier everywhere, and have
> to put down some serious money. (eg. travis-ci has a limit of 120
> minutes per build; the gitlab free tier is 1000 minutes/month)

So it does seem quite relevant to see how CI works out with regard to
the platform we use, and in particular how we can keep parts of the CI
on private systems if it turns out necessary for staying in free tiers.

-- 
David Kastrup