Re: [RFC PATCH 8/8] tests/docker: Selective line reading by python script

[Daniel P . Berrangé]

On Wed, Jul 27, 2022 at 11:20:56PM +0200, Philippe Mathieu-Daudé wrote:
> +Erik/Daniel
> 
> On Wed, Jul 27, 2022 at 6:37 PM Lucas Mateus Castro(alqotel)
> <lucas.araujo@eldorado.org.br> wrote:
> >
> > Building some images failed on ppc64le because the dockerfile tried to
> > install some packages that are only available in x86 and arm64, to solve
> > this while still having those packages be available in those architectures
> > a comment was put before the installation command to instruct the python
> > script into ignoring those lines for some architectures (in this case
> > ppc64le)
> >
> > Overall I'm not a big fan of the way I solved this problem, so I'd like
> > to know if anyone has a better way to make these dockerfilse work in
> > PPC64LE.
> >
> > For context the base images used here are available in PPC64LE but some
> > of the packages installed are not (in alpine's case it's XEN, which is
> > only available to x86 and ARM), so this patch create a ignore_list which
> > is set on a per-architecture basis, and any packages in a dockerfile in
> > this ignore_list will not be copied to the temporary dockerfile used in
> > the docker command.
> 
> Shouldn't this be done on lcitool side?
> (https://gitlab.com/libvirt/libvirt-ci/-/tree/master/lcitool)

Yes, to fix this properly we'll need to identify which packages are
architecture specific, and create separate 'RUN' commands that skip
those packages on arches in question. Doable, but not entirely easy.

> 
> > Signed-off-by: Lucas Mateus Castro(alqotel) <lucas.araujo@eldorado.org.br>
> > ---
> >  tests/docker/docker.py                 | 15 ++++++++++++---
> >  tests/docker/dockerfiles/alpine.docker |  2 ++
> >  2 files changed, 14 insertions(+), 3 deletions(-)
> >
> > diff --git a/tests/docker/docker.py b/tests/docker/docker.py
> > index d0af2861b8..9b962d1c78 100755
> > --- a/tests/docker/docker.py
> > +++ b/tests/docker/docker.py
> > @@ -14,6 +14,7 @@
> >  import os
> >  import sys
> >  import subprocess
> > +import platform
> >  import json
> >  import hashlib
> >  import atexit
> > @@ -207,8 +208,15 @@ def _read_qemu_dockerfile(img_name):
> >
> >  def _dockerfile_preprocess(df):
> >      out = ""
> > +    ignore_list = []
> >      for l in df.splitlines():
> > -        if len(l.strip()) == 0 or l.startswith("#"):
> > +        if len(l.strip()) == 0:
> > +            continue
> > +        if l.startswith("#"):
> > +            if len(l.split()) >= 3:
> > +                if l.split()[1] == "ignore":
> > +                    if platform.processor() in l.split()[2].split(','):
> > +                        ignore_list += l.split()[3].split(',')
> >              continue
> >          from_pref = "FROM qemu/"
> >          if l.startswith(from_pref):
> > @@ -219,7 +227,8 @@ def _dockerfile_preprocess(df):
> >              inlining = _read_qemu_dockerfile(l[len(from_pref):])
> >              out += _dockerfile_preprocess(inlining)
> >              continue
> > -        out += l + "\n"
> > +        if not any(x in l.split() for x in ignore_list):
> > +            out += l + "\n"
> >      return out
> >
> >
> > @@ -330,7 +339,7 @@ def build_image(self, tag, docker_dir, dockerfile,
> >          tmp_df = tempfile.NamedTemporaryFile(mode="w+t",
> >                                               encoding='utf-8',
> >                                               dir=docker_dir, 
> > suffix=".docker")
> > -        tmp_df.write(dockerfile)
> > +        tmp_df.write(_dockerfile_preprocess(dockerfile))
> >
> >          if user:
> >              uid = os.getuid()
> > diff --git a/tests/docker/dockerfiles/alpine.docker 
> > b/tests/docker/dockerfiles/alpine.docker
> > index 2943a99730..5cec46d8f2 100644
> > --- a/tests/docker/dockerfiles/alpine.docker
> > +++ b/tests/docker/dockerfiles/alpine.docker
> > @@ -6,6 +6,8 @@
> >
> >  FROM docker.io/library/alpine:edge
> >
> > +# Lines to by ignored when this file is read by the python script
> > +# ignore ppc64le,ppc64 xen-dev
> >  RUN apk update && \
> >      apk upgrade && \
> >      apk add \
> > --
> > 2.25.1
> >
> 

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|