Re: [PATCH] gitlab-ci: split clang-user to avoid timeout

[Richard Henderson]

On 11/4/22 09:32, Philippe Mathieu-Daudé wrote:
> +Richard
>
> On 3/11/22 22:23, Stefan Hajnoczi wrote:
> > GitLab CI times out when the clang-user job takes over 1 hour. Split it
> > into parts that check various architectures.
> >
> > An alternative is to have one job per architecture but that clutters the
> > pipeline view and maybe there is some sharing when multiple targets are
> > built at once.
> >
> > Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
> > ---
> >   .gitlab-ci.d/buildtest-template.yml | 11 +++++++++++
> >   .gitlab-ci.d/buildtest.yml          | 18 +++++++++---------
> >   2 files changed, 20 insertions(+), 9 deletions(-)
> >
> > diff --git a/.gitlab-ci.d/buildtest-template.yml 
> > b/.gitlab-ci.d/buildtest-template.yml
> > index 73ecfabb8d..38b055e139 100644
> > --- a/.gitlab-ci.d/buildtest-template.yml
> > +++ b/.gitlab-ci.d/buildtest-template.yml
> > @@ -81,3 +81,14 @@
> >       - du -chs ${CI_PROJECT_DIR}/avocado-cache
> >     variables:
> >       QEMU_JOB_AVOCADO: 1
> > +
> > +.clang-user-template:
> > +  extends: .native_build_job_template
> > +  needs:
> > +    job: amd64-debian-user-cross-container
> > +  variables:
> > +    IMAGE: debian-all-test-cross
> > +    CONFIGURE_ARGS: --cc=clang --cxx=clang++ --disable-system
> > +      --extra-cflags=-fsanitize=undefined
> > +      --extra-cflags=-fno-sanitize-recover=undefined
> > +    MAKE_CHECK_ARGS: check-unit check-tcg
> > diff --git a/.gitlab-ci.d/buildtest.yml b/.gitlab-ci.d/buildtest.yml
> > index 6c05c46397..116fce4e8f 100644
> > --- a/.gitlab-ci.d/buildtest.yml
> > +++ b/.gitlab-ci.d/buildtest.yml
> > @@ -323,16 +323,16 @@ clang-system:
> >         ppc-softmmu s390x-softmmu
> >       MAKE_CHECK_ARGS: check-qtest check-tcg
> > -clang-user:
> > -  extends: .native_build_job_template
> > -  needs:
> > -    job: amd64-debian-user-cross-container
> > +# clang-user takes too long so split it into parts
> > +clang-user-part1:
> > +  extends: .clang-user-template
> >     variables:
> > -    IMAGE: debian-all-test-cross
> > -    CONFIGURE_ARGS: --cc=clang --cxx=clang++ --disable-system
> > -      
> > --target-list-exclude=microblazeel-linux-user,aarch64_be-linux-user,i386-linux-user,m68k-linux-user,mipsn32el-linux-user,xtensaeb-linux-user
>
> We can exclude these targets which are a bit redundant:
>
>
> armeb-linux-user,mips64-linux-user,mipsel-linux-user,mipsn32-linux-user,ppc64-linux-user,sh4-linux-user,sparc-linux-user,riscv32-linux-user
>
> Alternatively, instead of using the exclude pattern, we can switch to including the 
> targets which do have tcg tests:
>
> $ ls -1 tests/tcg/
> Makefile.target
> README
> aarch64
> alpha
> arm
> cris
> hexagon
> hppa
> i386
> loongarch64
> m68k
> minilib
> mips
> multiarch
> nios2
> openrisc
> ppc
> ppc64
> ppc64le
> riscv64
> s390x
> sh4
> sparc64
> tricore
> x86_64
> xtensa
>
> Although we have 'multiarch' :/

If we're talking about tests, this runs on debian-all-test-cross, which has fewer 
cross-compilers than that.

However, the main purpose of clang-user is to make sure that stuff *builds* with clang, as 
opposed to gcc, which is where we've seen most problems in the past.  So we do want as 
much coverage across targets/*/ as possible, even if cross-compilers for tests are not 
available.

I agree that we can drop some redundancy, like aarch64_be, armbe, mips{set}, riscv32, 
which have no remarkable difference in linux-user/.  But be careful of e.g. ppc64 vs 
ppc64le and sparc vs sparc64 which have very different ABIs.

Perhaps an interesting split would be those guests supported by debian-all-test-cross, for 
which we build + test, and the others, for which we build only.


r~