Re: [RFC PATCH v3 1/1] gitlab: Use plain docker in container-template.yml

[Daniel P . Berrangé]

On Mon, Feb 27, 2023 at 06:12:30PM -0300, Fabiano Rosas wrote:
> Alex Bennée <alex.bennee@linaro.org> writes:
> 
> > Fabiano Rosas <farosas@suse.de> writes:
> >
> >> Our dockerfiles no longer reference layers from other qemu images so
> >> we can now use 'docker build' on them.
> >>
> >> Also reinstate the caching that was disabled due to bad interactions
> >> with certain runners. See commit 6ddc3dc7a8 ("tests/docker: don't use
> >> BUILDKIT in GitLab either"). We now believe those issues to be fixed.
> >>
> >> The COMMON_TAG needed to be fixed for the caching to work. The
> >> docker.py script was not using the variable, but constructing the
> >> correct URL directly.
> >>
> >> Signed-off-by: Fabiano Rosas <farosas@suse.de>
> >> ---
> >>  .gitlab-ci.d/container-template.yml | 9 ++++-----
> >>  1 file changed, 4 insertions(+), 5 deletions(-)
> >>
> >> diff --git a/.gitlab-ci.d/container-template.yml 
> >> b/.gitlab-ci.d/container-template.yml
> >> index c434b9c8f3..519b8a9482 100644
> >> --- a/.gitlab-ci.d/container-template.yml
> >> +++ b/.gitlab-ci.d/container-template.yml
> >> @@ -6,17 +6,16 @@
> >>      - docker:dind
> >>    before_script:
> >>      - export TAG="$CI_REGISTRY_IMAGE/qemu/$NAME:latest"
> >> -    - export COMMON_TAG="$CI_REGISTRY/qemu-project/qemu/$NAME:latest"
> >> +    - export COMMON_TAG="$CI_REGISTRY/qemu-project/qemu/qemu/$NAME:latest"
> >>      - apk add python3
> >>      - docker info
> >>      - docker login $CI_REGISTRY -u "$CI_REGISTRY_USER" -p 
> >> "$CI_REGISTRY_PASSWORD"
> >>    script:
> >>      - echo "TAG:$TAG"
> >>      - echo "COMMON_TAG:$COMMON_TAG"
> >> -    - ./tests/docker/docker.py --engine docker build
> >> -          -t "qemu/$NAME" -f "tests/docker/dockerfiles/$NAME.docker"
> >> -          -r $CI_REGISTRY/qemu-project/qemu
> >> -    - docker tag "qemu/$NAME" "$TAG"
> >> +    - docker build --tag "$TAG" --cache-from "$TAG" --cache-from 
> >> "$COMMON_TAG"
> >> +      --build-arg BUILDKIT_INLINE_CACHE=1
> >> +      -f "tests/docker/dockerfiles/$NAME.docker" "."
> >
> > I wonder why this doesn't injest a bunch of context. If I run:
> >
> >   docker build --cache-from 
> > registry.gitlab.com/stsquad/qemu/qemu/debian-alpha-cross --build-arg 
> > BUILDKIT_INLINE_CACHE=1  --build-arg USER=alex --build-arg UID=1000 -t qemu
> >   /debian-alpha-cross -f 
> > "/home/alex/lsrc/qemu.git/tests/docker/dockerfiles/debian-alpha-cross.docker"
> >  .
> >
> > it attempts to bring my entire build directory in as build context. This
> > is why we use the - < docker form in the Makefile.
> >
> 
> I only see that without DOCKER_BUILDKIT=1. With the variable set it does
> like in the CI jobs. I presume it is being set automatically by gitlab,
> but we could add it to the script to be explicit.
> 
> >>      - docker push "$TAG"
> >>    after_script:
> >>      - docker logout
> >
> > So what I don't understand is if I do:
> >
> >   docker pull registry.gitlab.com/stsquad/qemu/qemu/debian-alpha-cross
> >   docker build --cache-from 
> > registry.gitlab.com/stsquad/qemu/qemu/debian-alpha-cross --build-arg
> >     BUILDKIT_INLINE_CACHE=1 -t qemu/debian-alpha-cross - <
> >     
> > /home/alex/lsrc/qemu.git/tests/docker/dockerfiles/debian-alpha-cross.docker
> >
> > I still see pretty much a full rebuild of the image.
> 
> I don't use docker and podman does not support caching. I have
> fresh-installed docker today and indeed it seems to not use the cache at
> every build. We're missing something.

Empirically in gitlab I see it working as desired after applying this
patch. My first job building alpine container does all commands:

  https://gitlab.com/berrange/qemu/-/jobs/3845066410

My second job building alpine container uses the cache:

  https://gitlab.com/berrange/qemu/-/jobs/3845340414

IME of using this in libvirt the caching almost always did its job as
expected in gitlab.

IOW I wouldn't worry too much about local behaviour possibly differing.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|