Re: "make check-acceptance" takes way too long

[Philippe Mathieu-Daudé]

On 7/30/21 5:12 PM, Peter Maydell wrote:
> "make check-acceptance" takes way way too long. I just did a run
> on an arm-and-aarch64-targets-only debug build and it took over
> half an hour, and this despite it skipping or cancelling 26 out
> of 58 tests!
> 
> I think that ~10 minutes runtime is reasonable. 30 is not;
> ideally no individual test would take more than a minute or so.
> 
> Output saying where the time went. The first two tests take
> more than 10 minutes *each*. I think a good start would be to find
> a way of testing what they're testing that is less heavyweight.

IIRC the KVM forum BoF, we suggested a test shouldn't take more than
60sec. But then it was borderline for some tests so we talked about
allowing 90-120sec, and more should be discussed and documented.

However it was never documented / enforced.

This seems to match my memory:

$ git grep 'timeout =' tests/acceptance/
tests/acceptance/avocado_qemu/__init__.py:440:    timeout = 900
tests/acceptance/boot_linux_console.py:99:    timeout = 90
tests/acceptance/boot_xen.py:26:    timeout = 90
tests/acceptance/linux_initrd.py:27:    timeout = 300
tests/acceptance/linux_ssh_mips_malta.py:26:    timeout = 150 # Not for
'configure --enable-debug --enable-debug-tcg'
tests/acceptance/machine_arm_canona1100.py:18:    timeout = 90
tests/acceptance/machine_arm_integratorcp.py:34:    timeout = 90
tests/acceptance/machine_arm_n8x0.py:20:    timeout = 90
tests/acceptance/machine_avr6.py:25:    timeout = 5
tests/acceptance/machine_m68k_nextcube.py:30:    timeout = 15
tests/acceptance/machine_microblaze.py:14:    timeout = 90
tests/acceptance/machine_mips_fuloong2e.py:18:    timeout = 60
tests/acceptance/machine_mips_loongson3v.py:18:    timeout = 60
tests/acceptance/machine_mips_malta.py:38:    timeout = 30
tests/acceptance/machine_ppc.py:14:    timeout = 90
tests/acceptance/machine_rx_gdbsim.py:22:    timeout = 30
tests/acceptance/machine_s390_ccw_virtio.py:24:    timeout = 120
tests/acceptance/machine_sparc64_sun4u.py:20:    timeout = 90
tests/acceptance/machine_sparc_leon3.py:15:    timeout = 60
tests/acceptance/migration.py:27:    timeout = 10
tests/acceptance/ppc_prep_40p.py:18:    timeout = 60
tests/acceptance/replay_kernel.py:34:    timeout = 120
tests/acceptance/replay_kernel.py:357:    timeout = 180
tests/acceptance/reverse_debugging.py:33:    timeout = 10
tests/acceptance/tcg_plugins.py:24:    timeout = 120

> 
>  (01/58) tests/acceptance/boot_linux.py:BootLinuxAarch64.test_virt_tcg_gicv2:
> PASS (629.74 s)
>  (02/58) tests/acceptance/boot_linux.py:BootLinuxAarch64.test_virt_tcg_gicv3:
> PASS (628.75 s)
>  (03/58) tests/acceptance/boot_linux.py:BootLinuxAarch64.test_virt_kvm:
> CANCEL: kvm accelerator does not seem to be available (1.18 s)

We could restrict these to one of the projects runners (x86 probably)
with something like:

  @skipUnless(os.getenv('X86_64_RUNNER_AVAILABLE'), '...')

>  (15/58) 
> tests/acceptance/boot_linux_console.py:BootLinuxConsole.test_arm_orangepi:
> PASS (4.86 s)
>  (16/58) 
> tests/acceptance/boot_linux_console.py:BootLinuxConsole.test_arm_orangepi_initrd:
> PASS (39.85 s)
>  (17/58) 
> tests/acceptance/boot_linux_console.py:BootLinuxConsole.test_arm_orangepi_sd:
> PASS (53.57 s)
>  (18/58) 
> tests/acceptance/boot_linux_console.py:BootLinuxConsole.test_arm_orangepi_bionic_20_08:
> SKIP: storage limited
>  (19/58) 
> tests/acceptance/boot_linux_console.py:BootLinuxConsole.test_arm_orangepi_uboot_netbsd9:
> SKIP: storage limited

I've been thinking about restricting them to my sdmmc-tree, but if
I don't send pull-req I won't test or catch other introducing
regressions. They respect the 60sec limit.

We could restrict some jobs to maintainers fork namespace, track
mainstream master branch and either run the pipelines when /master
is updated or regularly
(https://docs.gitlab.com/ee/ci/pipelines/schedules.html)
but them if the maintainer becomes busy / idle / inactive we
similarly won't catch regressions in mainstream.

Anyway Daniel already studied the problem and send a RFC but we
ignored it:
https://www.mail-archive.com/qemu-devel@nongnu.org/msg761087.html

Maybe worth continuing the discussion there?