Re: [RFC PATCH v3 1/1] gitlab: Use plain docker in container-template.yml

[Fabiano Rosas]

Fabiano Rosas <farosas@suse.de> writes:

> Alex Bennée <alex.bennee@linaro.org> writes:
>
>> Fabiano Rosas <farosas@suse.de> writes:
>>
>>> Our dockerfiles no longer reference layers from other qemu images so
>>> we can now use 'docker build' on them.
>>>
>>> Also reinstate the caching that was disabled due to bad interactions
>>> with certain runners. See commit 6ddc3dc7a8 ("tests/docker: don't use
>>> BUILDKIT in GitLab either"). We now believe those issues to be fixed.
>>>
>>> The COMMON_TAG needed to be fixed for the caching to work. The
>>> docker.py script was not using the variable, but constructing the
>>> correct URL directly.
>>>
>>> Signed-off-by: Fabiano Rosas <farosas@suse.de>
>>> ---
>>>  .gitlab-ci.d/container-template.yml | 9 ++++-----
>>>  1 file changed, 4 insertions(+), 5 deletions(-)
>>>
>>> diff --git a/.gitlab-ci.d/container-template.yml 
>>> b/.gitlab-ci.d/container-template.yml
>>> index c434b9c8f3..519b8a9482 100644
>>> --- a/.gitlab-ci.d/container-template.yml
>>> +++ b/.gitlab-ci.d/container-template.yml
>>> @@ -6,17 +6,16 @@
>>>      - docker:dind
>>>    before_script:
>>>      - export TAG="$CI_REGISTRY_IMAGE/qemu/$NAME:latest"
>>> -    - export COMMON_TAG="$CI_REGISTRY/qemu-project/qemu/$NAME:latest"
>>> +    - export COMMON_TAG="$CI_REGISTRY/qemu-project/qemu/qemu/$NAME:latest"
>>>      - apk add python3
>>>      - docker info
>>>      - docker login $CI_REGISTRY -u "$CI_REGISTRY_USER" -p 
>>> "$CI_REGISTRY_PASSWORD"
>>>    script:
>>>      - echo "TAG:$TAG"
>>>      - echo "COMMON_TAG:$COMMON_TAG"
>>> -    - ./tests/docker/docker.py --engine docker build
>>> -          -t "qemu/$NAME" -f "tests/docker/dockerfiles/$NAME.docker"
>>> -          -r $CI_REGISTRY/qemu-project/qemu
>>> -    - docker tag "qemu/$NAME" "$TAG"
>>> +    - docker build --tag "$TAG" --cache-from "$TAG" --cache-from 
>>> "$COMMON_TAG"
>>> +      --build-arg BUILDKIT_INLINE_CACHE=1
>>> +      -f "tests/docker/dockerfiles/$NAME.docker" "."
>>
>> I wonder why this doesn't injest a bunch of context. If I run:
>>
>>   docker build --cache-from 
>> registry.gitlab.com/stsquad/qemu/qemu/debian-alpha-cross --build-arg 
>> BUILDKIT_INLINE_CACHE=1  --build-arg USER=alex --build-arg UID=1000 -t qemu
>>   /debian-alpha-cross -f 
>> "/home/alex/lsrc/qemu.git/tests/docker/dockerfiles/debian-alpha-cross.docker"
>>  .
>>
>> it attempts to bring my entire build directory in as build context. This
>> is why we use the - < docker form in the Makefile.
>>
>
> I only see that without DOCKER_BUILDKIT=1. With the variable set it does
> like in the CI jobs. I presume it is being set automatically by gitlab,
> but we could add it to the script to be explicit.
>
>>>      - docker push "$TAG"
>>>    after_script:
>>>      - docker logout
>>
>> So what I don't understand is if I do:
>>
>>   docker pull registry.gitlab.com/stsquad/qemu/qemu/debian-alpha-cross
>>   docker build --cache-from 
>> registry.gitlab.com/stsquad/qemu/qemu/debian-alpha-cross --build-arg
>>     BUILDKIT_INLINE_CACHE=1 -t qemu/debian-alpha-cross - <
>>     
>> /home/alex/lsrc/qemu.git/tests/docker/dockerfiles/debian-alpha-cross.docker
>>
>> I still see pretty much a full rebuild of the image.
>
> I don't use docker and podman does not support caching. I have
> fresh-installed docker today and indeed it seems to not use the cache at
> every build. We're missing something.
>
> Sometimes it works:
>

Oops, sorry about the long lines, here it is again:

===
$ docker system prune -a -f
Deleted build cache objects:
xzbxmzaib3a8s0ufetop5ikhi
0ce9qln4ipd2vgf9xw9to0fdb
se4hq3rce3lad20t9sqqnubob

Total reclaimed space: 5.845kB

$ docker images -a
REPOSITORY   TAG       IMAGE ID   CREATED   SIZE

$ DOCKER_BUILDKIT=1 docker build --tag
registry.gitlab.com/farosas/qemu/qemu/debian-amd64:latest --cache-from
registry.gitlab.com/farosas/qemu/qemu/debian-amd64:latest --build-arg
BUILDKIT_INLINE_CACHE=1 -f debian-amd64.docker .

[+] Building 57.8s (12/12) FINISHED
 => [internal] load build definition from debian-amd64.docker
 => => transferring dockerfile: 5.90kB
 => [internal] load .dockerignore
 => => transferring context: 2B
 => [internal] load metadata for docker.io/library/debian:11-slim
 => importing cache manifest from 
registry.gitlab.com/farosas/qemu/qemu/debian-amd64:latest
 => [1/6] FROM
docker.io/library/debian:11-slim@sha256:8eaee63a5ea83744e62d5bf88e7d472d7f19b5feda3bfc6a2304cc074f269269
 => CACHED [2/6] RUN export DEBIAN_FRONTEND=noninteractive &&     apt-get 
update &&     apt-get install -y eatmydata
&&     eatmydata apt-get dist-upgrade -y &&     eatmydata apt-get i
 => CACHED [3/6] RUN DEBIAN_FRONTEND=noninteractive eatmydata   apt install -y 
--no-install-recommends   cscope 
global  linux-headers-amd64
 => CACHED [4/6] RUN git clone https://github.com/luigirizzo/netmap.git 
/usr/src/netmap
 => CACHED [5/6] RUN cd /usr/src/netmap && git checkout v11.3
 => CACHED [6/6] RUN cd /usr/src/netmap/LINUX && ./configure --no-drivers 
--no-apps --kernel-dir=$(ls -d
/usr/src/linux-headers-*-amd64) && make install
 => => pulling 
sha256:bb263680fed18eecdc67f885094df6f589bafc19004839d7fdf141df236a61aa
 => => pulling 
sha256:f63092d28e024c119d6a4e257d0c62bb2117be93605f1f2363b1d9cb8fc9dc6c
 => => pulling 
sha256:30fcedea9de9f22da43e20a79b40516c6e4a55e70eea401acacca18912e86769
 => => pulling 
sha256:6f498daa6d20055a31e4681ec878e86129c3c1eaf5cf40f7ed87631accef1906
 => => pulling 
sha256:79b1b0d7bce9a16b4a01e2055cd402d93e04efded50e2ffcf6be0d7c67e4895a
 => => pulling 
sha256:250b927c5ce2d6d3e9d24bf085f649ce951966c9ec9c9e00ab8292dfb02c8b17
 => exporting to image
 => => exporting layers
 => => writing image 
sha256:3663223c0a31c2be678c2fe110ba284b80b8d04d2d3290c14be7a4d7320f8a67
 => => naming to registry.gitlab.com/farosas/qemu/qemu/debian-amd64:latest
 => exporting cache
 => => preparing build cache for export
===

>
> There are issues pointing to some general unreliability of the cache:
>
> https://github.com/moby/buildkit/issues/2274
> https://github.com/moby/buildkit/issues/1981
>
> What is your docker version? Mine is 20.10.17-ce and Gitlab's is
> 23.0.1. The issues above are on 20.10.11.