qemu-devel
[Top][All Lists]
Advanced

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [RFC PATCH-for-5.2] gitlab-ci: Do not automatically run Avocado inte


From: Wainer dos Santos Moschetta
Subject: Re: [RFC PATCH-for-5.2] gitlab-ci: Do not automatically run Avocado integration tests anymore
Date: Fri, 27 Nov 2020 16:36:43 -0300
User-agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.4.0

Hi Phil,

On 11/27/20 2:41 PM, Philippe Mathieu-Daudé wrote:
We lately realized that the Avocado framework was not designed
to be regularly run on CI environments. Therefore, as of 5.2
we deprecate the gitlab-ci jobs using Avocado. To not disrupt
current users, it is possible to keep the current behavior by
setting the QEMU_CI_INTEGRATION_JOBS_PRE_5_2_RELEASE variable
(see [*]).


Could you please detail where have you seen the avocado based tests failing that much to justify their removal of CI?

See below some data that I generated from the pipelines for https://gitlab.com/qemu-project/qemu/. It seems that jobs which failed due timeout are marked as 'skip' and that explain why some pipelines don't have a list of failed jobs.

---

Failed 21 out of 99
Pipeline -- 221617704 (staging)
check-patch
Pipeline -- 219276598 (master)
pages
Pipeline -- 218121424 (master)
Pipeline -- 217995873 (staging)
acceptance-system-fedora
Pipeline -- 217995872 (master)
build-tci
Pipeline -- 217551771 (v5.2.0-rc2)
Pipeline -- 217503505 (master)
Pipeline -- 217362429 (master)
Pipeline -- 217328707 (master)
check-crypto-old-nettle
Pipeline -- 216770735 (staging)
acceptance-system-fedora
check-patch
Pipeline -- 215772908 (master)
Pipeline -- 215715025 (staging)
check-crypto-old-gcrypt
Pipeline -- 215715024 (master)
check-crypto-old-nettle
Pipeline -- 214944520 (master)
acceptance-system-debian
Pipeline -- 214460230 (v5.2.0-rc1)
Pipeline -- 214438601 (master)
check-crypto-only-gnutls
Pipeline -- 214273938 (master)
Pipeline -- 214183970 (master)
Pipeline -- 214140305 (master)
Pipeline -- 213892224 (master)
Pipeline -- 213871132 (master)
 From now on, using these jobs (or adding new tests to them)
is strongly discouraged.

Tests based on Avocado will be ported to new job schemes during
the next releases, with better documentation and templates.

[*] 
https://docs.gitlab.com/ee/ci/variables/README.html#create-a-custom-variable-in-the-ui

Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
---
  .gitlab-ci.yml | 9 +++++++++
  1 file changed, 9 insertions(+)

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index d0173e82b16..2674407cd13 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -66,6 +66,15 @@ include:
      - cd build
      - python3 -c 'import json; r = json.load(open("tests/results/latest/results.json")); [print(t["logfile"]) for t in 
r["tests"] if t["status"] not in ("PASS", "SKIP", "CANCEL")]' | xargs cat
      - du -chs ${CI_PROJECT_DIR}/avocado-cache
+  rules:
+  # As of QEMU 5.2, Avocado is not yet ready to run in CI environments, 
therefore
+  # the jobs based on this template are not run automatically (except if the 
user
+  # explicitly sets the QEMU_CI_INTEGRATION_JOBS_PRE_5_2_RELEASE environment
+  # variable). Adding new jobs on top of this template is strongly discouraged.
+  - if: $QEMU_CI_INTEGRATION_JOBS_PRE_5_2_RELEASE == null
+    when: manual
+    allow_failure: true
+  - when: always
build-system-ubuntu:
    <<: *native_build_job_definition


The script I used to generate the above metrics:

$ pip install --user python-gitlab

$ cat gitlab-stats
#!/bin/env python3

import gitlab

gl = gitlab.Gitlab('http://gitlab.com')
qemu_project = gl.projects.get(11167699)
pipelines = qemu_project.pipelines.list()

total=0
failed=[]
for pipeline in qemu_project.pipelines.list(page=1, per_page=300):
    if pipeline.status == 'running':
        continue
    elif pipeline.status == 'failed':
        failed.append(pipeline)
    total += 1

print("Failed %d out of %d" % (len(failed), total))
for pipeline in failed:
    print('Pipeline -- %d (%s)' % (pipeline.id, pipeline.ref))
    jobs_failed=[]
    for job in pipeline.jobs.list():
        if job.status == 'failed':
            print(job.name)




reply via email to

[Prev in Thread] Current Thread [Next in Thread]