Re: [PATCH 2/2] gitlab: don't run CI jobs by default on push to user forks

[Cornelia Huck]

On Thu, Aug 12 2021, Daniel P. Berrangé <berrange@redhat.com> wrote:

> Currently pushes to user forks run the same set of build / test jobs as
> pushes to the main repo. This results in creation of 120+ individual
> jobs. While it is useful for subsystem maintainers, and even regular
> contributors to be able to run the full set of jobs, it is wasteful to
> run all of them all the time.
>
> In addition, with the expected change in GitLab to set a finite CI
> minute allowance on all users we need to be much more conservative.
>
> This patch thus sets up rules so that no CI jobs will run by default
> on push, except
>
>  - Pushes to branch name prefix 'staging' on 'qemu-project' repo
>    run full set of jobs
>  - Pushes to 'master' branch, only run jobs needed to publish
>    the website.
>
> In any pushes to users forks, CI is now strictly opt-in. The pipeline
> will always be created by every singe job will be marked manual. You
> can then use the web UI to start individual jobs.

Is a random user notified in some way that a pipeline with jobs that can
be triggered manually has been created?

> For more convenience there are three levels of increasing comprehensive
> CI job sets that can be chosen between
>
>  - Gating - run exactly the same as pushes to 'staging' branch
>             (except for jobs needing custom runners)
>
>    Push to 'ci-gating-xxx' branch, or set env QEMU_CI_GATING=1
>
>  - Full - same as 'Gating', except that acceptance tests don't
>           get run. This is equivalent to historical CI behaviour
>         for pushes to user forks.
>
>    Push to 'ci-full-xxx' branch, or set env QEMU_CI_FULL=1
>
>  - Minimal - a significantly cut down set of jobs to get a
>              decent sanity check of builds without burning
>            massive amounts of CI time.
>
>    Push to 'ci-min-xxx' branch, or set env QEMU_CI=1
>
> The minimal job set covers:
>
>   * Fedora, CentOS, Ubuntu & Debian system emulator builds
>     to cover all arch targets.
>   * Linux user static build
>   * Cross build i386 (to identify any 32-bit build issues)
>   * Cross build s390x (to identify any big endian build issues)
>   * Containers needed for the above
>   * Cirrus CI FreeBSD 12 and macOS 11 (to identify non-Linux issues)
>   * Simple checks - python unittests, DCO check, checkpatch.pl, etc
>
> This gives about 30 jobs instead of 120 from the "Full" group. It
> is likely reasonable to cut the minimal set down even more, as IMHO
> there are too many system emulator jobs there.

Where do you think we should start to cut them down? Limit the set of
tested arch targets to the most common ones?

Generally speaking, this makes sense; but I think we have different
situations which need different kinds of testing, and we should make it
as easy as possible to run the right set of tests.

(a) an individual contributor is doing some changes

In that case, I assume (hope?) that the contributor has actually
compiled the code for the relevant targets and has done some manual
testing. Running acceptance tests locally would also be good, or things
like iotests or check-tcg, when applicable.

(b) a subsystem maintainer is queuing some changes

Some more comprehensive automated testing, and likely some
subsystem-specific testing on top (like testing that requires
specialized hardware, manual interventions, etc.) Since the introduction
of gitlab-ci, I had relied on what you call the 'full' set to be run
automatically when I push to my staging branch, plus my manual
testing. Prior to that, I relied on patchew and ran tests
locally. Drawbacks of that: it hogs my development machine, and it is
easy to forget something.

If I pull from a sub-submaintainer, I assume that everything is already
in good order, and only do some very light sanity/integration
checking. Although we're hardly doing subsystem pull request nowadays.

(c) a subsystem maintainer is preparing a pull request

Ideally, that should run the 'gating' set, to eliminate needless bounces
of the pull request; plus some subsystem-specific manual testing on
top. In practice, the 'full' set might be good enough.

Couple of ideas from my side:
- include some scripts/make targets that guide an individual contributor
  to run a good subset of automated tests locally
- define a 'ci-subsystem' set that covers usual pain points for a
  subsystem
- have subsystem maintainers run a pull req test in the qemu-project
  context (i.e. using extra CI minutes that the project may have), or
  put them on a special list on subsystem maintainers so they can use
  more minutes