On Thu, Jun 25, 2020 at 12:14:33PM +0200, Thomas Huth wrote:
On 22/06/2020 17.33, Daniel P. Berrangé wrote:
We have a number of container images in tests/docker/dockerfiles
that are intended to provide well defined environments for doing
test builds. We want our CI system to use these containers too.
This introduces builds of all of them as the first stage in the
CI, so that the built containers are available for later build
jobs. The containers are setup to use the GitLab container
registry as the cache, so we only pay the penalty of the full
build when the dockerfiles change. The main qemu-project/qemu
repo is used as a second cache, so that users forking QEMU will
see a fast turnaround time on their CI jobs.
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
---
.gitlab-ci.d/containers.yml | 248 ++++++++++++++++++++++++++++++++++++
.gitlab-ci.yml | 3 +
2 files changed, 251 insertions(+)
create mode 100644 .gitlab-ci.d/containers.yml
diff --git a/.gitlab-ci.d/containers.yml b/.gitlab-ci.d/containers.yml
new file mode 100644
index 0000000000..ea1edbb196
--- /dev/null
+++ b/.gitlab-ci.d/containers.yml
@@ -0,0 +1,248 @@
+
+
+.container_job_template: &container_job_definition
+ image: docker:stable
+ stage: containers
+ services:
+ - docker:dind
+ before_script:
+ - export TAG="$CI_REGISTRY_IMAGE/ci-$NAME:latest"
+ - export COMMON_TAG="$CI_REGISTRY/qemu-project/qemu/ci-$NAME:latest"
+ - docker info
+ - docker login registry.gitlab.com -u "$CI_REGISTRY_USER" -p
"$CI_REGISTRY_PASSWORD"
+ script:
+ - docker pull "$TAG" || docker pull "$COMMON_TAG" || true
+ - sed -i -e "s,FROM qemu:,FROM $CI_REGISTRY_IMAGE/ci-,"
tests/docker/dockerfiles/$NAME.docker
+ - docker build --cache-from "$TAG" --cache-from "$COMMON_TAG" --tag "$TAG" -f
"tests/docker/dockerfiles/$NAME.docker" tests/docker/dockerfiles
+ - docker push "$TAG"
+ after_script:
+ - docker logout
.gitlab-ci.d/edk2.yml uses a "changes" rule to only run the pipeline if
something really has been changed. Could you use something similar here?
E.g.:
rules:
- changes:
- .gitlab-ci.d/containers.yml
- tests/docker/*
- tests/docker/dockerfiles/*
?
If the OS distro base image changes, we'll never pick it up with that
kind of filtering. For the main gitlab.com/qemu-project/qemu you
could configure a nightly/weekly/whatever job to force rebuild on a
periodic basis to pick up base image changes. The downside of this
is that any users who fork qemu won't have that periodic job and so
will be testing their work against potentially outdated content.
Having said all that, I'm not 100% convinced I'm actually picking
up changed base images right now anyway, given our use of caching.
It is possible that I would need todo an explict "docker pull" of
the base image to force it to trigger a refresh othrewise I have
a feeling we're always cached.