[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[PATCH v3 16/22] tests/docker: add script for automating container refre
From: |
Daniel P . Berrangé |
Subject: |
[PATCH v3 16/22] tests/docker: add script for automating container refresh |
Date: |
Thu, 13 May 2021 10:55:13 +0100 |
This introduces
https://gitlab.com/libvirt/libvirt-ci
as a git submodule at tests/docker/libvirt-ci
This submodule only needs to be checked out when needing to re-generate
the files in tests/docker/dockerfiles.
When a new build pre-requisite is needed for QEMU, it should be added to
the libvirt-ci project 'qemu.yml' file, and the submodule updated to the
new commit. The 'make docker-refresh' target will then re-create all the
dockerfiles with updated package lists. This ensures that all the
containers get exactly the same build pre-requisite packages installed.
It also facilitates the addition of containers targetting new distros
or updating existing containers to new versions of the same distro,
where packages might have been renamed.
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
---
.gitmodules | 3 ++
docs/devel/testing.rst | 15 ++++++--
tests/docker/Makefile.include | 10 ++++++
tests/docker/dockerfiles-refresh.py | 56 +++++++++++++++++++++++++++++
tests/docker/libvirt-ci | 1 +
5 files changed, 83 insertions(+), 2 deletions(-)
create mode 100755 tests/docker/dockerfiles-refresh.py
create mode 160000 tests/docker/libvirt-ci
diff --git a/.gitmodules b/.gitmodules
index 08b1b48a09..f78c63219b 100644
--- a/.gitmodules
+++ b/.gitmodules
@@ -64,3 +64,6 @@
[submodule "roms/vbootrom"]
path = roms/vbootrom
url = https://gitlab.com/qemu-project/vbootrom.git
+[submodule "tests/docker/libvirt-ci"]
+ path = tests/docker/libvirt-ci
+ url = https://gitlab.com/libvirt/libvirt-ci
diff --git a/docs/devel/testing.rst b/docs/devel/testing.rst
index 1da4c4e4c4..c159032e04 100644
--- a/docs/devel/testing.rst
+++ b/docs/devel/testing.rst
@@ -372,8 +372,19 @@ Along with many other images, the ``centos8`` image is
defined in a Dockerfile
in ``tests/docker/dockerfiles/``, called ``centos8.docker``. ``make
docker-help``
command will list all the available images.
-To add a new image, simply create a new ``.docker`` file under the
-``tests/docker/dockerfiles/`` directory.
+Most of the existing Dockerfiles were written by hand, simply by creating a
+a new ``.docker`` file under the ``tests/docker/dockerfiles/`` directory.
+This has led to an inconsistent set of packages being present across the
+different containers. Thus going forward, QEMU is aiming to automatically
+generate the Dockerfiles using the tools provided by the libvirt-ci project
+
+ https://gitlab.com/libvirt/libvirt-ci
+
+In that project, there is a master ``qemu.yml`` file defining the list of
+build pre-requisites, that is processed together with the ``mappings.yml``
+file to produce the distro specific list of package names. The automatically
+generated Dockerfiles are produced by ``tests/docker/dockerfiles-refresh.py``
+which is wired up to the ``make docker-refresh`` target.
A ``.pre`` script can be added beside the ``.docker`` file, which will be
executed before building the image under the build context directory. This is
diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
index 9f464cb92c..03e722b838 100644
--- a/tests/docker/Makefile.include
+++ b/tests/docker/Makefile.include
@@ -27,6 +27,8 @@ DOCKER_TESTS := $(notdir $(shell \
ENGINE := auto
DOCKER_SCRIPT=$(SRC_PATH)/tests/docker/docker.py --engine $(ENGINE)
+REFRESH_SCRIPT=$(SRC_PATH)/tests/docker/dockerfiles-refresh.py
+LCITOOL=$(SRC_PATH)/tests/docker/libvirt-ci/lcitool
TESTS ?= %
IMAGES ?= %
@@ -199,6 +201,7 @@ docker:
@echo ' docker-image: Build all images.'
@echo ' docker-image-IMAGE: Build image "IMAGE".'
@echo ' docker-run: For manually running a "TEST" with
"IMAGE".'
+ @echo ' docker-refresh: Re-generate dockerfiles after updating
libvirt-ci'
@echo
@echo 'Available container images:'
@echo ' $(DOCKER_IMAGES)'
@@ -281,3 +284,10 @@ docker-run-%:
docker-clean:
$(call quiet-command, $(DOCKER_SCRIPT) clean)
+
+docker-refresh:
+ @if [ ! -f $(LCITOOL) ] ; \
+ then \
+ cd $(SRC_PATH) && git submodule update --init
tests/docker/libvirt-ci ; \
+ fi
+ $(call quiet-command, $(REFRESH_SCRIPT) $(LCITOOL) $(SRC_PATH))
diff --git a/tests/docker/dockerfiles-refresh.py
b/tests/docker/dockerfiles-refresh.py
new file mode 100755
index 0000000000..7f59ffbc59
--- /dev/null
+++ b/tests/docker/dockerfiles-refresh.py
@@ -0,0 +1,56 @@
+#!/usr/bin/python3
+#
+# Re-generate container recipes
+#
+# This script uses the "lcitool" available from
+#
+# https://gitlab.com/libvirt/libvirt-ci
+#
+# Copyright (c) 2020 Red Hat Inc.
+#
+# This work is licensed under the terms of the GNU GPL, version 2
+# or (at your option) any later version. See the COPYING file in
+# the top-level directory.
+
+import sys
+import os
+import os.path
+import subprocess
+
+if len(sys.argv) != 3:
+ print("syntax: %s PATH-TO-LCITOOL SRC-ROOT" % sys.argv[0], file=sys.stderr)
+ sys.exit(1)
+
+lcitool_path=sys.argv[1]
+src_root=sys.argv[2]
+
+def atomic_write(filename, content):
+ dst = os.path.join(src_root, "tests", "docker", "dockerfiles", filename)
+ try:
+ with open(dst + ".tmp", "w") as fp:
+ print(content, file=fp, end="")
+ os.replace(dst + ".tmp", dst)
+ except Exception as ex:
+ os.unlink(dst + ".tmp")
+ raise
+
+def generate_image(filename, host, cross=None, trailer=None):
+ print("Generate %s" % filename)
+ args = [lcitool_path, "dockerfile"]
+ if cross is not None:
+ args.extend(["--cross", cross])
+ args.extend([host, "qemu"])
+ lcitool=subprocess.run(args, capture_output=True)
+
+ if lcitool.returncode != 0:
+ raise Exception("Failed to generate %s: %s" % (filename, lcitool.stderr))
+
+ content = lcitool.stdout.decode("utf8")
+ if trailer is not None:
+ content += trailer
+ atomic_write(filename, content)
+
+try:
+ pass
+except Exception as ex:
+ print(str(ex), file=sys.stderr)
diff --git a/tests/docker/libvirt-ci b/tests/docker/libvirt-ci
new file mode 160000
index 0000000000..1c5d87ecd2
--- /dev/null
+++ b/tests/docker/libvirt-ci
@@ -0,0 +1 @@
+Subproject commit 1c5d87ecd2283614a8b0c31cead0b6d7883afd28
--
2.31.1
- [PATCH v3 09/22] tests/docker: fix mistakes in ubuntu package lists, (continued)
- [PATCH v3 10/22] tests/docker: remove mingw packages from Fedora, Daniel P . Berrangé, 2021/05/13
- [PATCH v3 11/22] tests/docker: expand centos8 package list, Daniel P . Berrangé, 2021/05/13
- [PATCH v3 12/22] tests/docker: expand fedora package list, Daniel P . Berrangé, 2021/05/13
- [PATCH v3 13/22] tests/docker: expand ubuntu1804 package list, Daniel P . Berrangé, 2021/05/13
- [PATCH v3 14/22] tests/docker: expand ubuntu2004 package list, Daniel P . Berrangé, 2021/05/13
- [PATCH v3 15/22] tests/docker: expand opensuse-leap package list, Daniel P . Berrangé, 2021/05/13
- [PATCH v3 16/22] tests/docker: add script for automating container refresh,
Daniel P . Berrangé <=
- [PATCH v3 18/22] tests/docker: auto-generate fedora with lcitool, Daniel P . Berrangé, 2021/05/13
- [PATCH v3 17/22] tests/docker: auto-generate centos8 with lcitool, Daniel P . Berrangé, 2021/05/13
- [PATCH v3 19/22] tests/docker: auto-generate ubuntu1804 with lcitool, Daniel P . Berrangé, 2021/05/13
- [PATCH v3 20/22] tests/docker: auto-generate ubuntu2004 with lcitool, Daniel P . Berrangé, 2021/05/13
- [PATCH v3 21/22] tests/docker: auto-generate opensuse-leap with lcitool, Daniel P . Berrangé, 2021/05/13
- [PATCH v3 22/22] tests/docker: remove ubuntu container, Daniel P . Berrangé, 2021/05/13
- Re: [PATCH v3 00/22] tests/docker: start using libvirt-ci's "lcitool" for dockerfiles, no-reply, 2021/05/13