Re: [PATCH 09/12] tests: add tuxrun baseline test to avocado

[Philippe Mathieu-Daudé]

On 15/2/23 20:25, Alex Bennée wrote:
> The TuxRun project (www.tuxrun.org) uses QEMU to run tests on a wide
> variety of kernel configurations on wide range of our emulated
> platforms. They publish a known good set of images at:
>
>    https://storage.tuxboot.com/
>
> to help with bisecting regressions in either the kernel, firmware or
> QEMU itself. The tests are pretty lightweight as they contain just a
> kernel with a minimal rootfs which boots a lot faster than most of the
> distros. In time they might be persuaded to version there known good
> baselines and we can then enable proper checksums.
>
> For a couple of tests we currently skip:
>
>    - mips64, a regression against previous stable release
>    - sh4, very unstable with intermittent oops
>
> Total run time: 340s (default) -> 890s (debug)
>
> Overall coverage rate (tested targets + disabled tests):
>    lines......: 16.1% (126894 of 789848 lines)
>    functions..: 20.6% (15954 of 77489 functions)
>    branches...: 9.3% (40727 of 439365 branches)
>
> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
> Cc: Anders Roxell <anders.roxell@linaro.org>
>
> ---
> v2
>    - renamed to tuxrun_baselines, update commit message
>    - add remaining targets
>    - add more metadata tags for the differences
>    - refactor tag code
>    - skip mips64 and sh4 tests in CI
>    - slightly increase delay for login
>    - include in MAINTAINERS
> ---
>   MAINTAINERS                       |   1 +
>   tests/avocado/tuxrun_baselines.py | 423 ++++++++++++++++++++++++++++++
>   2 files changed, 424 insertions(+)
>   create mode 100644 tests/avocado/tuxrun_baselines.py
>
> diff --git a/MAINTAINERS b/MAINTAINERS
> index fd54c1f140..be100272b3 100644
> --- a/MAINTAINERS
> +++ b/MAINTAINERS
> @@ -3760,6 +3760,7 @@ F: scripts/ci/
>   F: tests/docker/
>   F: tests/vm/
>   F: tests/lcitool/
> +F: tests/avocado/tuxrun_baselines.py
>   F: scripts/archive-source.sh
>   F: docs/devel/testing.rst
>   W: https://gitlab.com/qemu-project/qemu/pipelines
> diff --git a/tests/avocado/tuxrun_baselines.py 
> b/tests/avocado/tuxrun_baselines.py
> new file mode 100644
> index 0000000000..30aaefc1d3
> --- /dev/null
> +++ b/tests/avocado/tuxrun_baselines.py
> @@ -0,0 +1,423 @@
> +# Functional test that boots known good tuxboot images the same way
> +# that tuxrun (www.tuxrun.org) does. This tool is used by things like
> +# the LKFT project to run regression tests on kernels.
> +#
> +# Copyright (c) 2023 Linaro Ltd.
> +#
> +# Author:
> +#  Alex Bennée <alex.bennee@linaro.org>
> +#
> +# SPDX-License-Identifier: GPL-2.0-or-later
> +
> +import os
> +import time
> +
> +from avocado import skip, skipIf
> +from avocado_qemu import QemuSystemTest
> +from avocado_qemu import exec_command, exec_command_and_wait_for_pattern
> +from avocado_qemu import wait_for_console_pattern
> +from avocado.utils import process
> +from avocado.utils.path import find_command
> +
> +class TuxRunBaselineTest(QemuSystemTest):

Better inherit from LinuxKernelTest, adding a zstd method there.

> +    """
> +    :avocado: tags=accel:tcg
> +    """
> +
> +    KERNEL_COMMON_COMMAND_LINE = 'printk.time=0'
> +    # Tests are ~10-40s, allow for --debug/--enable-gcov overhead
> +    timeout = 100
> +
> +    def get_tag(self, tagname, default=None):
> +        """
> +        Get the metadata tag or return the default.
> +        """
> +        utag = self._get_unique_tag_val(tagname)
> +        print(f"{tagname}/{default} -> {utag}")
> +        if utag:
> +            return utag
> +
> +        return default
> +
> +    def setUp(self):
> +        super().setUp()
> +
> +        # We need zstd for all the tuxrun tests
> +        # See https://github.com/avocado-framework/avocado/issues/5609
> +        zstd = find_command('zstd', False)
> +        if zstd is False:
> +            self.cancel('Could not find "zstd", which is required to '
> +                        'decompress rootfs')
> +        self.zstd = zstd
> +
> +        # Process the TuxRun specific tags, most machines work with
> +        # reasonable defaults but we sometimes need to tweak the
> +        # config. To avoid open coding everything we store all these
> +        # details in the metadata for each test.
> +
> +        # The tuxboot tag matches the root directory
> +        self.tuxboot = self.get_tag('tuxboot')
> +
> +        # Most Linux's use ttyS0 for their serial port
> +        self.console = self.get_tag('console', "ttyS0")
> +
> +        # Does the machine shutdown QEMU nicely on "halt"
> +        self.shutdown = self.get_tag('shutdown')
> +
> +        # The name of the kernel Image file
> +        self.image = self.get_tag('image', "Image")
> +
> +        # The block device drive type
> +        self.drive = self.get_tag('drive', "virtio-blk-device")
> +
> +        self.root = self.get_tag('root', "vda")
> +
> +        # Occasionally we need extra devices to hook things up
> +        self.extradev = self.get_tag('extradev')
> +
> +    def wait_for_console_pattern(self, success_message, vm=None):
> +        wait_for_console_pattern(self, success_message,
> +                                 failure_message='Kernel panic - not syncing',
> +                                 vm=vm)

(we'd then inherit this method from LinuxKernelTest)

Whichever outcome you rather:
Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>