Re: [PATCH v3 24/30] gitlab: add acceptance testing to system builds

[Alex Bennée]

Thomas Huth <thuth@redhat.com> writes:

> On 26/06/2020 20.13, Alex Bennée wrote:
>> As part of migrating things from Travis to GitLab add the acceptance
>> tests. To do this:
>> 
>>    - rename system1 to system-ubuntu-main
>>    - rename system2 to system-fedora-misc
>>    - split into build/check/acceptance
>>    - remove -j from check stages
>>    - use artifacts to save build stage
>>    - add post acceptance template and use
>> 
>> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
>> Message-Id: <20200622143204.12921-16-alex.bennee@linaro.org>
>> 
>> ---
>> v2
>>    - updated with danp's docker changes
>>    - use needs instead of dependancies
>>    - touch all the build files to prevent rebuild
>> ---
>>   .gitlab-ci.yml | 66 +++++++++++++++++++++++++++++++++++++++++++++++---
>>   .travis.yml    | 23 ------------------
>>   2 files changed, 63 insertions(+), 26 deletions(-)
>> 
>> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
>> index a7abc55a5c6..5ae8130bd1a 100644
>> --- a/.gitlab-ci.yml
>> +++ b/.gitlab-ci.yml
>> @@ -1,8 +1,12 @@
>> +# Currently we have two build stages after our containers are built:
>> +#  - build (for traditional build and test or first stage build)
>> +#  - test (for test stages, using build artefacts from a build stage)
>>   stages:
>>     - containers
>>     - containers-layer2
>>     - containers-layer3
>>     - build
>> +  - test
>>   
>>   include:
>>     - local: '/.gitlab-ci.d/edk2.yml'
>> @@ -24,26 +28,82 @@ include:
>>           ../configure --enable-werror $CONFIGURE_ARGS ;
>>         fi
>>       - make -j"$JOBS"
>> -    - make -j"$JOBS" $MAKE_CHECK_ARGS
>> +    - if test -n "$MAKE_CHECK_ARGS";
>> +      then
>> +        make $MAKE_CHECK_ARGS ;
>> +      fi
>> +
>> +.native_test_job_template: &native_test_job_definition
>> +  stage: test
>> +  image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
>> +  script:
>> +    - cd build
>> +    - find . -type f -exec touch {} +
>> +    - make $MAKE_CHECK_ARGS
>> +
>> +.post_acceptance_template: &post_acceptance
>> +  after_script:
>> +    - python3 -c 'import json; r = 
>> json.load(open("tests/results/latest/results.json")); [print(t["logfile"]) 
>> for t in r["tests"] if t["status"] not in ("PASS", "SKIP")]' | xargs cat
>> +    - du -chs $HOME/avocado/data/cache
>>   
>> -build-system1:
>> +build:system-ubuntu-main:
>>     <<: *native_build_job_definition
>>     variables:
>>       IMAGE: ubuntu2004
>>       TARGETS: aarch64-softmmu alpha-softmmu cris-softmmu hppa-softmmu 
>> lm32-softmmu
>>         moxie-softmmu microblazeel-softmmu mips64el-softmmu m68k-softmmu 
>> ppc-softmmu
>>         riscv64-softmmu sparc-softmmu
>> +  artifacts:
>> +    paths:
>> +      - build
>> +
>> +check:system-ubuntu-main:
>> +  <<: *native_test_job_definition
>> +  needs:
>> +    - job: build:system-ubuntu-main
>> +      artifacts: true
>> +  variables:
>> +    IMAGE: ubuntu2004
>>       MAKE_CHECK_ARGS: check
>>   
>> -build-system2:
>> +acceptance:system-ubuntu-main:
>> +  <<: *native_test_job_definition
>> +  needs:
>> +    - job: build:system-ubuntu-main
>> +      artifacts: true
>> +  variables:
>> +    IMAGE: ubuntu2004
>> +    MAKE_CHECK_ARGS: check-acceptance
>> +
>> +build:system-fedora-alt:
>>     <<: *native_build_job_definition
>>     variables:
>>       IMAGE: fedora
>>       TARGETS: tricore-softmmu unicore32-softmmu microblaze-softmmu 
>> mips-softmmu
>>         riscv32-softmmu s390x-softmmu sh4-softmmu sparc64-softmmu 
>> x86_64-softmmu
>>         xtensa-softmmu nios2-softmmu or1k-softmmu
>> +  artifacts:
>> +    paths:
>> +      - build
>> +
>> +check:system-fedora-alt:
>> +  <<: *native_test_job_definition
>> +  needs:
>> +    - job: build:system-fedora-alt
>> +      artifacts: true
>> +  variables:
>> +    IMAGE: fedora
>>       MAKE_CHECK_ARGS: check
>>   
>> +check:system-fedora-alt:
>> +  <<: *native_test_job_definition
>> +  needs:
>> +    - job: build:system-fedora-alt
>> +      artifacts: true
>> +  variables:
>> +    IMAGE: fedora
>> +    MAKE_CHECK_ARGS: check-acceptance
>
> Why is Ubuntu "-main" and "Fedora "-alt" ? ... that does not make sense 
> to me.

In a previous revision I had one do MAIN_SOFTMMU_TARGETS and therefor
the other was the alternate ones.

> We also might want to rework the list of targets. To speed up the 
> testing, I originally omitted some targets like sh4eb-softmmu which did 
> not seem very interesting, but now that we add more and more builds in 
> parallel, we could maybe split the two system targets into three or even 
> four instead, and then add these targets, too. It would also be nice to 
> have some tests with "centos8" and a debian container, too.
>
> And "rx-softmmu" is also still missing in the target list.
>
> Ok, it's quite a bit of change that still needs to be done here ... 
> maybe that's rather something for a separate patch later.

It's getting tricky with --target-list and target-list-exclude because
the run times are getting quite long although the splitting helps.

>
>   Thomas


-- 
Alex Bennée