Re: [PATCH v2 2/2] GitLab Gating CI: initial set of jobs, documentation and scripts

[Daniel P . Berrangé]

On Wed, Jul 08, 2020 at 10:46:57PM -0400, Cleber Rosa wrote:
> This is a mapping of Peter's "remake-merge-builds" and
> "pull-buildtest" scripts, gone through some updates, adding some build
> option and removing others.
> 
> The jobs currently cover the machines that the QEMU project owns, and that
> are setup and ready to run jobs:
> 
>  - Ubuntu 18.04 on S390x
>  - Ubuntu 20.04 on aarch64
> 
> During the development of this set of jobs, the GitLab CI was tested
> with many other architectures, including ppc64, s390x and aarch64,
> along with the other OSs (not included here):
> 
>  - Fedora 30
>  - FreeBSD 12.1
> 
> More information can be found in the documentation itself.
> 
> Signed-off-by: Cleber Rosa <crosa@redhat.com>
> ---
>  .gitlab-ci.d/gating.yml                | 146 +++++++++++++++++

AFAIK, the jobs in this file just augment what is already defined
in the main .gitlab-ci.yml. Also since we're providing setup info
for other people to configure custom runners, these jobs are usable
for non-gating CI scenarios too.

IOW, the jobs in this file happen to be usable for gating, but they
are not the only gating jobs, and can be used for non-gating reasons.

This is a complicated way of saying that gating.yml is not a desirable
filename, so I'd suggest splitting it in two and having these files
named based on what their contents is, rather than their use case:

   .gitlab-ci.d/runners-s390x.yml
   .gitlab-ci.d/runners-aarch64.yml

The existing jobs in .gitlab-ci.yml could possibly be moved into
a .gitlab-ci.d/runners-shared.yml file for consistency.

>  .gitlab-ci.yml                         |   1 +
>  docs/devel/testing.rst                 | 147 +++++++++++++++++
>  scripts/ci/setup/build-environment.yml | 217 +++++++++++++++++++++++++
>  scripts/ci/setup/gitlab-runner.yml     |  72 ++++++++
>  scripts/ci/setup/inventory             |   2 +
>  scripts/ci/setup/vars.yml              |  13 ++
>  7 files changed, 598 insertions(+)
>  create mode 100644 .gitlab-ci.d/gating.yml
>  create mode 100644 scripts/ci/setup/build-environment.yml
>  create mode 100644 scripts/ci/setup/gitlab-runner.yml
>  create mode 100644 scripts/ci/setup/inventory
>  create mode 100644 scripts/ci/setup/vars.yml
> 
> diff --git a/.gitlab-ci.d/gating.yml b/.gitlab-ci.d/gating.yml
> new file mode 100644
> index 0000000000..5562df5708
> --- /dev/null
> +++ b/.gitlab-ci.d/gating.yml
> @@ -0,0 +1,146 @@
> +variables:
> +  GIT_SUBMODULE_STRATEGY: recursive
> +
> +# All ubuntu-18.04 jobs should run successfully in an environment
> +# setup by the scripts/ci/setup/build-environment.yml task
> +# "Install basic packages to build QEMU on Ubuntu 18.04/20.04"
> +ubuntu-18.04-s390x-all-linux-static:
> + tags:
> + - ubuntu_18.04
> + - s390x
> + rules:
> + - if: '$CI_COMMIT_REF_NAME == "staging"'

I think I'd make it more flexible in particular to allow multiple
branches. For example I have multiple subsystems and have separate
branches for each.

This could be as simple as allowing a regex prefix

  - if: '$CI_COMMIT_REF_NAME =~ /^staging/'




> diff --git a/scripts/ci/setup/build-environment.yml 
> b/scripts/ci/setup/build-environment.yml
> new file mode 100644
> index 0000000000..89b35386c7
> --- /dev/null
> +++ b/scripts/ci/setup/build-environment.yml
> @@ -0,0 +1,217 @@
> +---
> +- name: Installation of basic packages to build QEMU
> +  hosts: all
> +  vars_files:
> +    - vars.yml
> +  tasks:
> +    - name: Install basic packages to build QEMU on Ubuntu 18.04/20.04
> +      apt:
> +        update_cache: yes
> +        # This matches the packages on 
> tests/docker/Dockerfiles/ubuntu1804.docker

I'd be inclined to actually use docker on the custom runners.

eg. instead of having separate physical machines or VMs for each
(distro, arch) pair, have a single host distro for the arch. Then
use docker to provide the build environment against each distro.

IOW, a RHEL-8 aarch64 host, running docker for ubuntu18.04, fedora30
etc.

That way we don't end up duplicating all these packages, and instead
can use  tests/docker/Dockerfiles/ubuntu1804.docker.  This ensures
that if a user needs to reproduce a build failure on their own local
aarch64 machine, they can run docker and get the exact same build
architecture.

It also has the benefit that we don't need to worry about how to
setup gitlab runners for every distro we care about. We only need to
do gitlab runner for the standard host distro, which spawns a pristine
throwaway docker env.

I appreciate this is a big change from what you've done in this patch
though, so don't consider this comment a blocker for initial merge.
I think we should do this as the long term strategy though. Essentially
for Linux builds, everything should always be container based.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|