Re: testing out Docker CI scripts?

[David Kastrup]

Han-Wen Nienhuys <address@hidden> writes:

> On Sat, Feb 22, 2020 at 3:06 PM David Kastrup <address@hidden> wrote:
>> > I would be interested in your feedback.
>> >
>> > For example, I think it could be used for validating the staging ->
>> > master push in the following way:
>> >
>> >  ( cd  lilypond ; git fetch origin ) && \
>> > sh test-git.sh --ubuntu lilypond/  origin/staging  && \
>> > sh test-git.sh --fedora lilypond/  origin/staging  && \
>> >  (cd lilypond && git push origin/staging:master)
>>
>> That's oversimplifying the staging->master push process a bit which goes
>> to considerable pain to make sure that its own copy of staging is still
>> part of the upstream staging branch before pushing the tested result.
>> That makes it possible to manually stop a bad staging commit from
>> reaching master even if it would compile: once you reset staging, no
>> already running Patchy process will override that decision with a
>> version of staging that has become stale.
>
> If there are multiple patchy processes, you'd hope they all come to
> the same conclusion.

Patchy processes don't reset staging.  Humans do.  But our various
patchies run on different platforms with different version libraries.
That actually has turned out helpful in discovering portability problems
at times.

> I think the code you are talking about is
>
>   
> https://github.com/gperciva/lilypond-extra/blob/master/patches/compile_lilypond_test/__init__.py#L467
>
> which doesn't look very advanced to me.

I haven't said anything about it being "very advanced" or impossible to
implement.  I have said that your proposed example is an
oversimplification of what we are doing.

> You can express it as a shell script like
>
>   git fetch origin &&
>   BEFORE=$(git rev-parse origin/staging) &&
>   ( ..test as above.. ) &&
>   git fetch origin &&
>   test "$BEFORE" = $(git rev-parse origin/staging) &&
>   git push origin origin/staging:master
>
>> This kind of double care is tricky enough to warrant its own
>> script/logic rather than rely on someone remembering all the details.
>
> I'm not suggesting someone remembers this. Instead, we can put the
> above commands in a shell script.
>
> I want to reiterate the advantages relative to patchy:
>
> 1) we get a reproducible test process, because everyone can use the
> same base images.

Which makes it less likely that we discover portability problems.  I am
not sure what problem you are trying to address here.

> 2) we can test against different configurations (Pango 1.44 vs. 1.36,
> GUILE 1.8 vs 2.2) simultaneously, which catches problems like the
> recent Pango one earlier.

That's definitely an advantage against our more haphazard setup now.  It
does come at the cost of _everyone_ (or the CI system) having to test
_all_ pertinent configurations rather than just a personal sampling if
it is supposed to increase the covered base.

-- 
David Kastrup
My replies have a tendency to cause friction.  To help mitigating
damage, feel free to forward problematic posts to me adding a subject
like "timeout 1d" (for a suggested timeout of 1 day) or "offensive".