Re: [PATCH v3 20/26] tests/avocado: disable sh4 rd2 tests on Gitlab

[Alex Bennée]

Alex Bennée <alex.bennee@linaro.org> writes:

> Running repeated invocations on a number of test boxes show a fairly
> high error rate:
>
>   $ retry.py -n 100 -c -- ./tests/venv/bin/avocado run 
> tests/avocado/boot_linux_console.py:BootLinuxConsole.test_sh4_r2d
>   retry.py called with ['./tests/venv/bin/avocado', 'run', 
> 'tests/avocado/boot_linux_console.py:BootLinuxConsole.test_sh4_r2d']
>   Results:
>   Run, Ret, Pass/Fail, Time, Total Pass, Total Run
>   ...
>   Results summary:
>   0: 94 times (94.00%), avg time 2.254 (0.00 varience/0.04 deviation)
>   1: 3 times (3.00%), avg time 1.837 (0.02 varience/0.14 deviation)
>   8: 3 times (3.00%), avg time 91.288 (0.02 varience/0.15 deviation)
>
> Examining the logs they fall into various categories of un-handled
> unaligned access by user space and unexpected FPU usage by the kernel
> which ultimately lead to the failure to reach the login prompt. This
> could be bugs in the translator that only get hit occasionally or just
> a flaky kernel - its hard to tell. To avoid these failures gating CI
> lets skip on GitLab.
>
> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
> Cc: Thomas Huth <thuth@redhat.com>
> Cc: Yoshinori Sato <ysato@users.sourceforge.jp>
> Cc: Magnus Damm <magnus.damm@gmail.com>
> ---
>  tests/avocado/boot_linux_console.py | 4 ++++
>  1 file changed, 4 insertions(+)
>
> diff --git a/tests/avocado/boot_linux_console.py 
> b/tests/avocado/boot_linux_console.py
> index eed4b49e6e..26eaf1a119 100644
> --- a/tests/avocado/boot_linux_console.py
> +++ b/tests/avocado/boot_linux_console.py
> @@ -1175,6 +1175,10 @@ def test_ppc_mac99(self):
>          self.vm.add_args('-M', 'graphics=off')
>          self.do_test_advcal_2018('15', tar_hash, 'invaders.elf')
>  
> +    # This test has a 6-10% failure rate on various hosts that look
> +    # like issues with a buggy kernel. As a result we don't want it
> +    # gating releases on Gitlab.
> +    @skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab')

I also forgot to add the import:

from avocado import skipIf

>      def test_sh4_r2d(self):
>          """
>          :avocado: tags=arch:sh4


-- 
Alex Bennée