Re: "make check-acceptance" takes way too long

[Alex Bennée]

Peter Maydell <peter.maydell@linaro.org> writes:

> "make check-acceptance" takes way way too long. I just did a run
> on an arm-and-aarch64-targets-only debug build and it took over
> half an hour, and this despite it skipping or cancelling 26 out
> of 58 tests!
>
> I think that ~10 minutes runtime is reasonable. 30 is not;
> ideally no individual test would take more than a minute or so.
>
> Output saying where the time went. The first two tests take
> more than 10 minutes *each*. I think a good start would be to find
> a way of testing what they're testing that is less heavyweight.
>
>  (01/58) tests/acceptance/boot_linux.py:BootLinuxAarch64.test_virt_tcg_gicv2:
> PASS (629.74 s)
>  (02/58) tests/acceptance/boot_linux.py:BootLinuxAarch64.test_virt_tcg_gicv3:
> PASS (628.75 s)
>  (03/58) tests/acceptance/boot_linux.py:BootLinuxAarch64.test_virt_kvm:
> CANCEL: kvm accelerator does not seem to be available (1.18 s)

For these tests which purport to exercise the various GIC configurations
I think we would be much better served by running kvm-unit-tests which
at least try and exercise all the features rather than rely on the side
effect of booting an entire OS.

>  (04/58) 
> tests/acceptance/boot_linux_console.py:BootLinuxConsole.test_aarch64_virt:
> PASS (3.53 s)
>  (05/58) 
> tests/acceptance/boot_linux_console.py:BootLinuxConsole.test_aarch64_xlnx_versal_virt:
> PASS (41.13 s)
>  (06/58) 
> tests/acceptance/boot_linux_console.py:BootLinuxConsole.test_arm_virt:
> PASS (5.22 s)
>  (07/58) 
> tests/acceptance/boot_linux_console.py:BootLinuxConsole.test_arm_emcraft_sf2:
> PASS (18.88 s)
>  (08/58) 
> tests/acceptance/boot_linux_console.py:BootLinuxConsole.test_arm_raspi2_uart0:
> PASS (11.30 s)
>  (09/58) 
> tests/acceptance/boot_linux_console.py:BootLinuxConsole.test_arm_raspi2_initrd:
> PASS (22.66 s)
>  (10/58) 
> tests/acceptance/boot_linux_console.py:BootLinuxConsole.test_arm_exynos4210_initrd:
> PASS (31.89 s)
>  (11/58) 
> tests/acceptance/boot_linux_console.py:BootLinuxConsole.test_arm_cubieboard_initrd:
> PASS (27.86 s)
>  (12/58) 
> tests/acceptance/boot_linux_console.py:BootLinuxConsole.test_arm_cubieboard_sata:
> PASS (27.19 s)
>  (13/58) 
> tests/acceptance/boot_linux_console.py:BootLinuxConsole.test_arm_quanta_gsj:
> SKIP: Test might timeout
>  (14/58) 
> tests/acceptance/boot_linux_console.py:BootLinuxConsole.test_arm_quanta_gsj_initrd:
> PASS (22.53 s)
>  (15/58) 
> tests/acceptance/boot_linux_console.py:BootLinuxConsole.test_arm_orangepi:
> PASS (4.86 s)
>  (16/58) 
> tests/acceptance/boot_linux_console.py:BootLinuxConsole.test_arm_orangepi_initrd:
> PASS (39.85 s)
>  (17/58) 
> tests/acceptance/boot_linux_console.py:BootLinuxConsole.test_arm_orangepi_sd:
> PASS (53.57 s)
>  (18/58) 
> tests/acceptance/boot_linux_console.py:BootLinuxConsole.test_arm_orangepi_bionic_20_08:
> SKIP: storage limited
>  (19/58) 
> tests/acceptance/boot_linux_console.py:BootLinuxConsole.test_arm_orangepi_uboot_netbsd9:
> SKIP: storage limited
>  (20/58) 
> tests/acceptance/boot_linux_console.py:BootLinuxConsole.test_aarch64_raspi3_atf:
> PASS (1.50 s)
>  (21/58) 
> tests/acceptance/boot_linux_console.py:BootLinuxConsole.test_arm_vexpressa9:
> PASS (10.74 s)
>  (22/58) 
> tests/acceptance/boot_linux_console.py:BootLinuxConsole.test_arm_ast2400_palmetto_openbmc_v2_9_0:
> PASS (39.43 s)
>  (23/58) 
> tests/acceptance/boot_linux_console.py:BootLinuxConsole.test_arm_ast2500_romulus_openbmc_v2_9_0:
> PASS (54.01 s)
>  (24/58) 
> tests/acceptance/boot_linux_console.py:BootLinuxConsole.test_arm_ast2600_debian:
> PASS (40.60 s)
>  (25/58) tests/acceptance/boot_xen.py:BootXen.test_arm64_xen_411_and_dom0:
> PASS (20.22 s)
>  (26/58) tests/acceptance/boot_xen.py:BootXen.test_arm64_xen_414_and_dom0:
> PASS (17.37 s)
>  (27/58) tests/acceptance/boot_xen.py:BootXen.test_arm64_xen_415_and_dom0:
> PASS (23.82 s)
>  (28/58) tests/acceptance/empty_cpu_model.py:EmptyCPUModel.test:
> CANCEL: No QEMU binary defined or found in the build tree (0.00 s)
>  (29/58) tests/acceptance/info_usernet.py:InfoUsernet.test_hostfwd:
> CANCEL: No QEMU binary defined or found in the build tree (0.00 s)
>  (30/58) 
> tests/acceptance/machine_arm_canona1100.py:CanonA1100Machine.test_arm_canona1100:
> PASS (0.20 s)
>  (31/58) 
> tests/acceptance/machine_arm_integratorcp.py:IntegratorMachine.test_integratorcp_console:
> SKIP: untrusted code
>  (32/58) 
> tests/acceptance/machine_arm_integratorcp.py:IntegratorMachine.test_framebuffer_tux_logo:
> SKIP: Python NumPy not installed
>  (33/58) tests/acceptance/machine_arm_n8x0.py:N8x0Machine.test_n800:
> SKIP: untrusted code
>  (34/58) tests/acceptance/machine_arm_n8x0.py:N8x0Machine.test_n810:
> SKIP: untrusted code
>  (35/58) 
> tests/acceptance/migration.py:Migration.test_migration_with_tcp_localhost:
> CANCEL: No QEMU binary defined or found in the build tree (0.00 s)
>  (36/58) tests/acceptance/migration.py:Migration.test_migration_with_unix:
> CANCEL: No QEMU binary defined or found in the build tree (0.00 s)
>  (37/58) tests/acceptance/migration.py:Migration.test_migration_with_exec:
> CANCEL: No QEMU binary defined or found in the build tree (0.00 s)
>  (38/58) 
> tests/acceptance/multiprocess.py:Multiprocess.test_multiprocess_aarch64:
> CANCEL: kvm accelerator does not seem to be available (0.06 s)
>  (39/58) 
> tests/acceptance/replay_kernel.py:ReplayKernelNormal.test_aarch64_virt:
> PASS (19.59 s)
>  (40/58) tests/acceptance/replay_kernel.py:ReplayKernelNormal.test_arm_virt:
> PASS (28.73 s)
>  (41/58) 
> tests/acceptance/replay_kernel.py:ReplayKernelNormal.test_arm_cubieboard_initrd:
> PASS (52.00 s)
>  (42/58) 
> tests/acceptance/replay_kernel.py:ReplayKernelNormal.test_arm_vexpressa9:
> PASS (25.69 s)
>  (43/58) 
> tests/acceptance/reverse_debugging.py:ReverseDebugging_AArch64.test_aarch64_virt:
> PASS (2.16 s)
>  (44/58) tests/acceptance/smmu.py:SMMU.test_smmu_noril: CANCEL: kvm
> accelerator does not seem to be available (0.90 s)
>  (45/58) tests/acceptance/smmu.py:SMMU.test_smmu_noril_passthrough:
> CANCEL: kvm accelerator does not seem to be available (0.70 s)
>  (46/58) tests/acceptance/smmu.py:SMMU.test_smmu_noril_nostrict:
> CANCEL: kvm accelerator does not seem to be available (1.02 s)
>  (47/58) tests/acceptance/smmu.py:SMMU.test_smmu_ril: CANCEL: kvm
> accelerator does not seem to be available (0.68 s)
>  (48/58) tests/acceptance/smmu.py:SMMU.test_smmu_ril_passthrough:
> CANCEL: kvm accelerator does not seem to be available (0.98 s)
>  (49/58) tests/acceptance/smmu.py:SMMU.test_smmu_ril_nostrict: CANCEL:
> kvm accelerator does not seem to be available (1.00 s)
>  (50/58) 
> tests/acceptance/tcg_plugins.py:PluginKernelNormal.test_aarch64_virt_insn:
> PASS (12.19 s)
>  (51/58) 
> tests/acceptance/tcg_plugins.py:PluginKernelNormal.test_aarch64_virt_insn_icount:
> PASS (12.35 s)
>  (52/58) 
> tests/acceptance/tcg_plugins.py:PluginKernelNormal.test_aarch64_virt_mem_icount:
> PASS (10.21 s)
>  (53/58) tests/acceptance/version.py:Version.test_qmp_human_info_version:
> CANCEL: No QEMU binary defined or found in the build tree (0.00 s)
>  (54/58) 
> tests/acceptance/virtio_check_params.py:VirtioMaxSegSettingsCheck.test_machine_types:
> SKIP: break multi-arch CI
>  (55/58) tests/acceptance/vnc.py:Vnc.test_no_vnc: CANCEL: No QEMU
> binary defined or found in the build tree (0.00 s)
>  (56/58) tests/acceptance/vnc.py:Vnc.test_no_vnc_change_password:
> CANCEL: No QEMU binary defined or found in the build tree (0.00 s)
>  (57/58) tests/acceptance/vnc.py:Vnc.test_change_password_requires_a_password:
> CANCEL: No QEMU binary defined or found in the build tree (0.00 s)
>  (58/58) tests/acceptance/vnc.py:Vnc.test_change_password: CANCEL: No
> QEMU binary defined or found in the build tree (0.00 s)
> RESULTS    : PASS 32 | ERROR 0 | FAIL 0 | SKIP 8 | WARN 0 | INTERRUPT
> 0 | CANCEL 18
> JOB TIME   : 1967.44 s
>
> thanks
> -- PMM


-- 
Alex Bennée