Re: [PATCH] gitlab-ci: split clang-user to avoid timeout

[Philippe Mathieu-Daudé]

+Richard

On 3/11/22 22:23, Stefan Hajnoczi wrote:
> GitLab CI times out when the clang-user job takes over 1 hour. Split it
> into parts that check various architectures.
>
> An alternative is to have one job per architecture but that clutters the
> pipeline view and maybe there is some sharing when multiple targets are
> built at once.
>
> Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
> ---
>   .gitlab-ci.d/buildtest-template.yml | 11 +++++++++++
>   .gitlab-ci.d/buildtest.yml          | 18 +++++++++---------
>   2 files changed, 20 insertions(+), 9 deletions(-)
>
> diff --git a/.gitlab-ci.d/buildtest-template.yml 
> b/.gitlab-ci.d/buildtest-template.yml
> index 73ecfabb8d..38b055e139 100644
> --- a/.gitlab-ci.d/buildtest-template.yml
> +++ b/.gitlab-ci.d/buildtest-template.yml
> @@ -81,3 +81,14 @@
>       - du -chs ${CI_PROJECT_DIR}/avocado-cache
>     variables:
>       QEMU_JOB_AVOCADO: 1
> +
> +.clang-user-template:
> +  extends: .native_build_job_template
> +  needs:
> +    job: amd64-debian-user-cross-container
> +  variables:
> +    IMAGE: debian-all-test-cross
> +    CONFIGURE_ARGS: --cc=clang --cxx=clang++ --disable-system
> +      --extra-cflags=-fsanitize=undefined
> +      --extra-cflags=-fno-sanitize-recover=undefined
> +    MAKE_CHECK_ARGS: check-unit check-tcg
> diff --git a/.gitlab-ci.d/buildtest.yml b/.gitlab-ci.d/buildtest.yml
> index 6c05c46397..116fce4e8f 100644
> --- a/.gitlab-ci.d/buildtest.yml
> +++ b/.gitlab-ci.d/buildtest.yml
> @@ -323,16 +323,16 @@ clang-system:
>         ppc-softmmu s390x-softmmu
>       MAKE_CHECK_ARGS: check-qtest check-tcg
>   
> -clang-user:
> -  extends: .native_build_job_template
> -  needs:
> -    job: amd64-debian-user-cross-container
> +# clang-user takes too long so split it into parts
> +clang-user-part1:
> +  extends: .clang-user-template
>     variables:
> -    IMAGE: debian-all-test-cross
> -    CONFIGURE_ARGS: --cc=clang --cxx=clang++ --disable-system
> -      
> --target-list-exclude=microblazeel-linux-user,aarch64_be-linux-user,i386-linux-user,m68k-linux-user,mipsn32el-linux-user,xtensaeb-linux-user

We can exclude these targets which are a bit redundant:


armeb-linux-user,mips64-linux-user,mipsel-linux-user,mipsn32-linux-user,ppc64-linux-user,sh4-linux-user,sparc-linux-user,riscv32-linux-user

Alternatively, instead of using the exclude pattern, we can switch to 
including the targets which do have tcg tests:

$ ls -1 tests/tcg/
Makefile.target
README
aarch64
alpha
arm
cris
hexagon
hppa
i386
loongarch64
m68k
minilib
mips
multiarch
nios2
openrisc
ppc
ppc64
ppc64le
riscv64
s390x
sh4
sparc64
tricore
x86_64
xtensa

Although we have 'multiarch' :/

We can also drop check-unit, but not sure if it saves much.

> -      --extra-cflags=-fsanitize=undefined 
> --extra-cflags=-fno-sanitize-recover=undefined
> -    MAKE_CHECK_ARGS: check-unit check-tcg
> +    TARGETS: 
> aarch64-linux-user,alpha-linux-user,armeb-linux-user,arm-linux-user,cris-linux-user,hexagon-linux-user,hppa-linux-user,loongarch64-linux-user,microblaze-linux-user,mips64el-linux-user,mips64-linux-user,mipsel-linux-user,mips-linux-user,mipsn32-linux-user
> +
> +clang-user-part2:
> +  extends: .clang-user-template
> +  variables:
> +    TARGETS: 
> nios2-linux-user,or1k-linux-user,ppc64le-linux-user,ppc64-linux-user,ppc-linux-user,riscv32-linux-user,riscv64-linux-user,s390x-linux-user,sh4eb-linux-user,sh4-linux-user,sparc32plus-linux-user,sparc64-linux-user,sparc-linux-user,x86_64-linux-user,xtensa-linux-user
>   
>   # Set LD_JOBS=1 because this requires LTO and ld consumes a large amount of memory.
>   # On gitlab runners, default value sometimes end up calling 2 lds 
> concurrently and