[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [PATCH 4/4] Jobs based on custom runners: add CentOS Stream 8
From: |
Cleber Rosa Junior |
Subject: |
Re: [PATCH 4/4] Jobs based on custom runners: add CentOS Stream 8 |
Date: |
Wed, 9 Jun 2021 16:37:35 -0400 |
On Tue, Jun 8, 2021 at 10:10 AM Cleber Rosa <crosa@redhat.com> wrote:
>
> This introduces three different parts of a job designed to run
> on a custom runner managed by Red Hat. The goals include:
>
> a) serve as a model for other organizations that want to onboard
> their own runners, with their specific platforms, build
> configuration and tests.
>
> b) bring awareness to the differences between upstream QEMU and the
> version available under CentOS Stream, which is "A preview of
> upcoming Red Hat Enterprise Linux minor and major releases.".
>
> c) becase of b), it should be easier to identify and reduce the gap
> between Red Hat's downstream and upstream QEMU.
>
> The components themselves to achieve this custom job are:
>
> 1) build environment configuration: documentation and a playbook for
> a base Enterprise Linux 8 system (also applicable to CentOS
> Stream), which other users can run on their system to get the
> environment suitable for building QEMU.
>
> 2) QEMU build configuration: how QEMU will be built to match, as
> closely as possible, the binaries built and packaged on CentOS
> stream 8.
>
> 3) job definition: GitLab CI jobs that will dispatch the build/test
> job to the machine specifically configured according to #1.
>
> Signed-off-by: Cleber Rosa <crosa@redhat.com>
> ---
> .gitlab-ci.d/custom-runners.yml | 29 ++++
> scripts/ci/org.centos/stream/README | 2 +
> scripts/ci/org.centos/stream/configure | 190 +++++++++++++++++++++++++
> scripts/ci/setup/build-environment.yml | 38 +++++
> 4 files changed, 259 insertions(+)
> create mode 100644 scripts/ci/org.centos/stream/README
> create mode 100755 scripts/ci/org.centos/stream/configure
>
> diff --git a/.gitlab-ci.d/custom-runners.yml b/.gitlab-ci.d/custom-runners.yml
> index 061d3cdfed..ee5143995e 100644
> --- a/.gitlab-ci.d/custom-runners.yml
> +++ b/.gitlab-ci.d/custom-runners.yml
> @@ -220,3 +220,32 @@ ubuntu-20.04-aarch64-notcg:
> - ../configure --disable-libssh --disable-tcg
> - make --output-sync -j`nproc`
> - make --output-sync -j`nproc` check V=1
> +
> +centos-stream-8-x86_64:
> + allow_failure: true
> + needs: []
> + stage: build
> + tags:
> + - centos_stream_8
> + - x86_64
> + rules:
> + - if: '$CI_COMMIT_BRANCH =~ /^staging/'
> + artifacts:
> + name: "$CI_JOB_NAME-$CI_COMMIT_REF_SLUG"
> + when: on_failure
> + expire_in: 7 days
> + paths:
> + - build/tests/results/latest/results.xml
> + - build/tests/results/latest/test-results
> + reports:
> + junit: build/tests/results/latest/results.xml
> + script:
> + - mkdir build
> + - cd build
> + - ../scripts/ci/org.centos/stream/configure
> + - make --output-sync -j`nproc`
> + - make --output-sync -j`nproc` check V=1
> + - make get-vm-images
> + # Only run tests that are either marked explicitly for KVM and x86_64
> + # or tests that are supposed to be valid for all targets
> + - ./tests/venv/bin/avocado run --job-results-dir=tests/results/
> --filter-by-tags-include-empty --filter-by-tags-include-empty-key -t
> accel:kvm,arch:x86_64 -- tests/acceptance/
> diff --git a/scripts/ci/org.centos/stream/README
> b/scripts/ci/org.centos/stream/README
> new file mode 100644
> index 0000000000..f99bda99b8
> --- /dev/null
> +++ b/scripts/ci/org.centos/stream/README
> @@ -0,0 +1,2 @@
> +This directory contains scripts for generating a build of QEMU that
> +closely matches the CentOS Stream builds of the qemu-kvm package.
> diff --git a/scripts/ci/org.centos/stream/configure
> b/scripts/ci/org.centos/stream/configure
> new file mode 100755
> index 0000000000..1e7207faec
> --- /dev/null
> +++ b/scripts/ci/org.centos/stream/configure
> @@ -0,0 +1,190 @@
> +#!/bin/sh -e
> +../configure \
> +--prefix="/usr" \
> +--libdir="/usr/lib64" \
> +--datadir="/usr/share" \
> +--sysconfdir="/etc" \
> +--interp-prefix=/usr/qemu-%M \
> +--localstatedir="/var" \
> +--docdir="/usr/share/doc" \
> +--libexecdir="/usr/libexec" \
> +--extra-ldflags="-Wl,--build-id -Wl,-z,relro -Wl,-z,now" \
> +--extra-cflags="-O2 -g -pipe -Wall -Werror=format-security
> -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions
> -fstack-protector-strong -grecord-gcc-switches
> -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1
> -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic
> -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection" \
> +--with-suffix="qemu-kvm" \
> +--firmwarepath=/usr/share/qemu-firmware \
> +--meson="/usr/bin/meson" \
> +--target-list="x86_64-softmmu" \
> +--block-drv-rw-whitelist=qcow2,raw,file,host_device,nbd,iscsi,rbd,blkdebug,luks,null-co,nvme,copy-on-read,throttle,gluster
> \
> +--audio-drv-list= \
> +--block-drv-ro-whitelist=vmdk,vhdx,vpc,https,ssh \
> +--with-coroutine=ucontext \
> +--with-git=git \
> +--tls-priority=@QEMU,SYSTEM \
> +--disable-attr \
> +--disable-auth-pam \
> +--disable-avx2 \
> +--disable-avx512f \
> +--disable-bochs \
> +--disable-brlapi \
> +--disable-bsd-user \
> +--disable-bzip2 \
> +--disable-cap-ng \
> +--disable-capstone \
> +--disable-cfi \
> +--disable-cfi-debug \
> +--disable-cloop \
> +--disable-cocoa \
> +--disable-coroutine-pool \
> +--disable-crypto-afalg \
> +--disable-curl \
> +--disable-curses \
> +--disable-debug-info \
> +--disable-debug-mutex \
> +--disable-debug-tcg \
> +--disable-dmg \
> +--disable-docs \
> +--disable-fdt \
> +--disable-fuse \
> +--disable-fuse-lseek \
> +--disable-gcrypt \
> +--disable-gio \
> +--disable-glusterfs \
> +--disable-gnutls \
> +--disable-gtk \
> +--disable-guest-agent \
> +--disable-guest-agent-msi \
> +--disable-hax \
> +--disable-hvf \
> +--disable-iconv \
> +--disable-jemalloc \
> +--disable-kvm \
> +--disable-libdaxctl \
> +--disable-libiscsi \
> +--disable-libnfs \
> +--disable-libpmem \
> +--disable-libssh \
> +--disable-libudev \
> +--disable-libusb \
> +--disable-libxml2 \
> +--disable-linux-aio \
> +--disable-linux-io-uring \
> +--disable-linux-user \
> +--disable-live-block-migration \
> +--disable-lto \
> +--disable-lzfse \
> +--disable-lzo \
> +--disable-malloc-trim \
> +--disable-membarrier \
> +--disable-modules \
> +--disable-module-upgrades \
> +--disable-mpath \
> +--disable-multiprocess \
> +--disable-netmap \
> +--disable-nettle \
> +--disable-numa \
> +--disable-opengl \
> +--disable-parallels \
> +--disable-pie \
> +--disable-pvrdma \
> +--disable-qcow1 \
> +--disable-qed \
> +--disable-qom-cast-debug \
> +--disable-rbd \
> +--disable-rdma \
> +--disable-replication \
> +--disable-rng-none \
> +--disable-safe-stack \
> +--disable-sanitizers \
> +--disable-sdl \
> +--disable-sdl-image \
> +--disable-seccomp \
> +--disable-smartcard \
> +--disable-snappy \
> +--disable-sparse \
> +--disable-spice \
> +--disable-strip \
> +--disable-system \
> +--disable-tcg \
> +--disable-tcmalloc \
> +--disable-tools \
> +--disable-tpm \
> +--disable-u2f \
> +--disable-usb-redir \
> +--disable-user \
> +--disable-vde \
> +--disable-vdi \
> +--disable-vhost-crypto \
> +--disable-vhost-kernel \
> +--disable-vhost-net \
> +--disable-vhost-scsi \
> +--disable-vhost-user \
> +--disable-vhost-user-blk-server \
> +--disable-vhost-vdpa \
> +--disable-vhost-vsock \
> +--disable-virglrenderer \
> +--disable-virtfs \
> +--disable-virtiofsd \
> +--disable-vnc \
> +--disable-vnc-jpeg \
> +--disable-vnc-png \
> +--disable-vnc-sasl \
> +--disable-vte \
> +--disable-vvfat \
> +--disable-werror \
> +--disable-whpx \
> +--disable-xen \
> +--disable-xen-pci-passthrough \
> +--disable-xfsctl \
> +--disable-xkbcommon \
> +--disable-zstd \
> +--enable-attr \
> +--enable-avx2 \
> +--enable-cap-ng \
> +--enable-capstone \
> +--enable-coroutine-pool \
> +--enable-curl \
> +--enable-debug-info \
> +--enable-docs \
> +--enable-gcrypt \
> +--enable-glusterfs \
> +--enable-gnutls \
> +--enable-guest-agent \
> +--enable-iconv \
> +--enable-kvm \
> +--enable-libiscsi \
> +--enable-libpmem \
> +--enable-libssh \
> +--enable-libusb \
> +--enable-libudev \
> +--enable-linux-aio \
> +--enable-lzo \
> +--enable-malloc-trim \
> +--enable-modules \
> +--enable-mpath \
> +--enable-numa \
> +--enable-opengl \
> +--enable-pie \
> +--enable-rbd \
> +--enable-rdma \
> +--enable-seccomp \
> +--enable-snappy \
> +--enable-smartcard \
> +--enable-spice \
> +--enable-system \
> +--enable-tcg \
> +--enable-tools \
> +--enable-tpm \
> +--enable-trace-backend=dtrace \
> +--enable-usb-redir \
> +--enable-virtiofsd \
> +--enable-vhost-kernel \
> +--enable-vhost-net \
> +--enable-vhost-user \
> +--enable-vhost-user-blk-server \
> +--enable-vhost-vdpa \
> +--enable-vhost-vsock \
> +--enable-vnc \
> +--enable-vnc-png \
> +--enable-vnc-sasl \
> +--enable-werror \
> +--enable-xkbcommon
> diff --git a/scripts/ci/setup/build-environment.yml
> b/scripts/ci/setup/build-environment.yml
> index 664f2f0519..b1e01b1025 100644
> --- a/scripts/ci/setup/build-environment.yml
> +++ b/scripts/ci/setup/build-environment.yml
> @@ -96,3 +96,41 @@
> when:
> - ansible_facts['distribution'] == 'Ubuntu'
> - ansible_facts['distribution_version'] == '20.04'
> +
> + - name: Install basic packages to build QEMU on EL8
> + dnf:
> + # This list of packages start with
> tests/docker/dockerfiles/centos8.docker
> + # but only include files that are common to all distro variants and
> present
> + # in the standard repos (no add-ons)
> + name:
> + - bzip2
> + - bzip2-devel
> + - dbus-daemon
> + - diffutils
> + - gcc
> + - gcc-c++
> + - genisoimage
> + - gettext
> + - git
> + - glib2-devel
> + - libaio-devel
> + - libepoxy-devel
> + - libgcrypt-devel
> + - lzo-devel
> + - make
> + - mesa-libEGL-devel
> + - nettle-devel
> + - nmap-ncat
> + - perl-Test-Harness
> + - pixman-devel
> + - python36
> + - rdma-core-devel
> + - spice-glib-devel
> + - spice-server
> + - systemtap-sdt-devel
> + - tar
> + - zlib-devel
> + state: present
> + when:
> + - ansible_facts['distribution_file_variety'] == 'RedHat'
> + - ansible_facts['distribution_version'] == '8'
> --
> 2.25.4
>
An apology and a heads up to reviewers: I left the CentOS Stream
specific repo/packages playbook unstaged, and the README should be
more informative about it. So here it is:
diff --git a/scripts/ci/org.centos/stream/README
b/scripts/ci/org.centos/stream/README
index f99bda99b8..a6f0566145 100644
--- a/scripts/ci/org.centos/stream/README
+++ b/scripts/ci/org.centos/stream/README
@@ -1,2 +1,11 @@
This directory contains scripts for generating a build of QEMU that
closely matches the CentOS Stream builds of the qemu-kvm package.
+
+To have your the environment ready to configure, build QEMU and run
+its tests, please:
+
+ * apply the generic "build-environment.yml" playbook located at
+ scripts/ci/setup
+
+ * apply the "build-environment.yml" in this directory, which is
+ specific to CentOS Stream.
diff --git a/scripts/ci/org.centos/stream/build-environment.yml
b/scripts/ci/org.centos/stream/build-environment.yml
new file mode 100644
index 0000000000..832bba8bc6
--- /dev/null
+++ b/scripts/ci/org.centos/stream/build-environment.yml
@@ -0,0 +1,51 @@
+---
+- name: Installation of extra packages to build QEMU
+ hosts: all
+ tasks:
+ - name: Extra check for CentOS Stream 8
+ lineinfile:
+ path: /etc/redhat-release
+ line: CentOS Stream release 8
+ state: present
+ check_mode: yes
+ register: centos_stream_8
+
+ - name: Enable PowerTools repo on CentOS Stream 8
+ ini_file:
+ path: /etc/yum.repos.d/CentOS-Stream-PowerTools.repo
+ section: powertools
+ option: enabled
+ value: "1"
+ when:
+ - ansible_facts['distribution'] == 'CentOS'
+ - ansible_facts['distribution_major_version'] == '8'
+ - centos_stream_8
+
+ - name: Install basic packages to build QEMU on CentOS Stream 8
+ dnf:
+ name:
+ - device-mapper-multipath-devel
+ - glusterfs-api-devel
+ - gnutls-devel
+ - libcap-ng-devel
+ - libcurl-devel
+ - libiscsi-devel
+ - libpmem-devel
+ - librados-devel
+ - librbd-devel
+ - libseccomp-devel
+ - libssh-devel
+ - libxkbcommon-devel
+ - meson
+ - ninja-build
+ - numactl-devel
+ - python3-sphinx
+ - redhat-rpm-config
+ - snappy-devel
+ - spice-server-devel
+ - systemd-devel
+ state: present
+ when:
+ - ansible_facts['distribution'] == 'CentOS'
+ - ansible_facts['distribution_major_version'] == '8'
+ - centos_stream_8
- [PATCH 0/4] Jobs based on custom runners: add CentOS Stream 8, Cleber Rosa, 2021/06/08
- [PATCH 1/4] block.c: fix compilation error on possible unitialized variable, Cleber Rosa, 2021/06/08
- [PATCH 2/4] Python QEMU utils: introduce a generic feature list, Cleber Rosa, 2021/06/08
- [PATCH 3/4] Acceptance Tests: introduce method to require a feature and option, Cleber Rosa, 2021/06/08
- [PATCH 4/4] Jobs based on custom runners: add CentOS Stream 8, Cleber Rosa, 2021/06/08
- Re: [PATCH 0/4] Jobs based on custom runners: add CentOS Stream 8, Willian Rampazzo, 2021/06/10