Re: [PATCH 2/2] gitlab: don't run CI jobs by default on push to user forks

[Philippe Mathieu-Daudé]

On 8/16/21 1:03 PM, Daniel P. Berrangé wrote:
> On Mon, Aug 16, 2021 at 12:44:02PM +0200, Cornelia Huck wrote:
>> On Thu, Aug 12 2021, Daniel P. Berrangé <berrange@redhat.com> wrote:
>>> The minimal job set covers:
>>>
>>>   * Fedora, CentOS, Ubuntu & Debian system emulator builds
>>>     to cover all arch targets.
>>>   * Linux user static build
>>>   * Cross build i386 (to identify any 32-bit build issues)
>>>   * Cross build s390x (to identify any big endian build issues)
>>>   * Containers needed for the above
>>>   * Cirrus CI FreeBSD 12 and macOS 11 (to identify non-Linux issues)
>>>   * Simple checks - python unittests, DCO check, checkpatch.pl, etc
>>>
>>> This gives about 30 jobs instead of 120 from the "Full" group. It
>>> is likely reasonable to cut the minimal set down even more, as IMHO
>>> there are too many system emulator jobs there.
>>
>> Where do you think we should start to cut them down? Limit the set of
>> tested arch targets to the most common ones?
> 
> Some of our targets are obviously much more important and
> frequently changed than others.  For contributors our goal is
> to mimimize breakage after patches are submitted. Most of our
> contributors changes will be well covered by x86-64 + aarch64
> alone. Other targets give varying degrees of extra benefit.
> 
> On the other hand the contributors are likely to have tested
> x86_64 or aarch64 themselves since that'll be their dev
> platform. So the benefit of CI is testing bits that they
> didnt bother to test.
> 
> No clear easy answer here, but I feel like we could benefit
> from classifying our target archs tier 1/2/3 and tailoring
> our default testing matrix accordingly.

[*]

> The other way to cut down the "minimal" set is to reduce
> the OS containers that we build. The jobs above end up
> requiring something like 8 container builds - we should
> try to cut this down to perhaps 2-3 container builds
> 
>> Generally speaking, this makes sense; but I think we have different
>> situations which need different kinds of testing, and we should make it
>> as easy as possible to run the right set of tests.
>>
>> (a) an individual contributor is doing some changes
>>
>> In that case, I assume (hope?) that the contributor has actually
>> compiled the code for the relevant targets and has done some manual
>> testing. Running acceptance tests locally would also be good, or things
>> like iotests or check-tcg, when applicable.
> 
> With my contributor hat on, I like GitLab CI to validate the platforms
> I always forget. Changes I do are 95% tested on Fedora x86_64. I have
> often broken stuff for non-Linux builds (Windows in particular), or
> have broken non-x86_64 target arches. CI lets me see this before
> sending patches.  Unfortunately this means I benefit most from the
> "full" set, but this won't be sustainable with limited CI minutes :-(

Hmmm somehow. What we said 2 years ago was "if a vendor is interested
in have QEMU working on a target, it should provide hardware to the
project, otherwise it can't be considered a tier 1 platform" [*].
Some did, but we have been wasting their resources doing almost nothing,
mostly because we were not ready / organized to use them. Hopefully this
might change soon.

> When I'm working on changing gitlab CI rules, then I burn loads of
> minutes which is especially troubling - limited CI minutes will make
> it very hard for me to debug future CI rule changes :-(

Which is why I stopped doing gitlab-related patches and reduced
my testing (or even worst for me, before I was simply doing
'git-push' and wait for the pipeline result, now I'm back
running 6 different scripts locally).

>> (c) a subsystem maintainer is preparing a pull request
>>
>> Ideally, that should run the 'gating' set, to eliminate needless bounces
>> of the pull request; plus some subsystem-specific manual testing on
>> top. In practice, the 'full' set might be good enough.
> 
> Yeah, the full/gating set is what I would thing subsys maintainers
> would want to use, to minimize risk that Peter's tests throw back
> the merge due to failure. The only difference of gating vs full
> is whether the acceptance tests run.

"run" or "pass"?

> Of course this relies on us being able to use GitLab for 100% of
> merge time gating. Cleber's custom runners recently enabled get
> us closer, but I think Peter still uses some other hardware
> outside of GitLab for some testing.

There is hope!