Re: [RFC PATCH] tests: add LKFT baseline test to avocado

[Philippe Mathieu-Daudé]

On 7/2/23 21:44, Alex Bennée wrote:
> The Linux Kernel Function Test (LKFT) project uses QEMU to test a wide
> variety of kernel configurations on wide range of our emulated
> platforms. They publish a known good set of images at:
>
>    https://storage.tuxboot.com/
>
> to help with bisecting regressions in either the kernel, firmware or
> QEMU itself. The tests are pretty lightweight as they contain just a
> kernel with a minimal rootfs which boots a lot faster than most of the
> distros. In time they might be persuaded to version there known good
> baselines and we can then enable proper checksums.
>
> Total run time: 140s
>
> Overall coverage rate:
>    lines......: 8.7% (96412 of 1106284 lines)
>    functions..: 10.8% (11515 of 106651 functions)
>    branches...: 8.3% (30685 of 370255 branches)
>
> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
> Cc: Anders Roxell <anders.roxell@linaro.org>
> ---
>   tests/avocado/lkft_baselines.py | 257 ++++++++++++++++++++++++++++++++
>   1 file changed, 257 insertions(+)
>   create mode 100755 tests/avocado/lkft_baselines.py

Do we want to skip the whole class on CI, or only some tests?

   @skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab')

> +class LKFTBaselineTest(QemuSystemTest):
> +    KERNEL_COMMON_COMMAND_LINE = 'printk.time=0 root=/dev/vda'
> +
> +    def setUp(self):
> +        super().setUp()
> +
> +        # We need zstd for all the lkft tests
> +        zstd = find_command('zstd', False)
> +        if zstd is False:
> +            self.cancel('Could not find "zstd", which is required to '
> +                        'decompress rootfs')
> +        self.zstd = zstd

Requested in upstream Avocado:
https://github.com/avocado-framework/avocado/issues/5609

> +    def fetch_lkft_assets(self, dt=None):
> +        """
> +        Fetch the LKFT assets. They are stored in a standard way so we
> +        use the per-test tags to fetch details.
> +        """
> +        base_url = f"https://storage.tuxboot.com/{self.lkft}/";
> +        kernel_image =  self.fetch_asset(base_url + self.image)
> +        disk_image_zst = self.fetch_asset(base_url + "rootfs.ext4.zst")

Hmm no hash verification.

> +    def test_arm64be(self):
> +        """
> +        :avocado: tags=arch:aarch64

This tag is useful:

           :avocado: tags=endian:big

> +        :avocado: tags=cpu:cortex-a57
> +        :avocado: tags=machine:virt
> +        :avocado: tags=lkft:arm64be
> +        :avocado: tags=console:ttyAMA0
> +        :avocado: tags=shutdown:nowait
> +        """
> +        self.common_lkft()


> +    def test_armv7(self):
> +        """
> +        :avocado: tags=arch:arm
> +        :avocado: tags=cpu:cortex-a15
> +        :avocado: tags=machine:virt
> +        :avocado: tags=lkft:armv7
> +        :avocado: tags=image:zImage
> +        :avocado: tags=console:ttyAMA0
> +        :avocado: tags=shutdown:nowait
> +        """
> +        self.common_lkft()
> +
> +    def test_armv7be(self):
> +        """
> +        :avocado: tags=arch:arm

           :avocado: tags=endian:big

> +        :avocado: tags=cpu:cortex-a15
> +        :avocado: tags=machine:virt
> +        :avocado: tags=lkft:armv7be
> +        :avocado: tags=image:zImage
> +        :avocado: tags=console:ttyAMA0
> +        :avocado: tags=shutdown:nowait
> +        """
> +        self.common_lkft()