Re: "make check-acceptance" takes way too long

[Alex Bennée]

Daniel P. Berrangé <berrange@redhat.com> writes:

> On Fri, Jul 30, 2021 at 04:12:27PM +0100, Peter Maydell wrote:
>> "make check-acceptance" takes way way too long. I just did a run
>> on an arm-and-aarch64-targets-only debug build and it took over
>> half an hour, and this despite it skipping or cancelling 26 out
>> of 58 tests!
>> 
>> I think that ~10 minutes runtime is reasonable. 30 is not;
>> ideally no individual test would take more than a minute or so.
>> 
>> Output saying where the time went. The first two tests take
>> more than 10 minutes *each*. I think a good start would be to find
>> a way of testing what they're testing that is less heavyweight.
>
> While there is certainly value in testing with a real world "full" guest
> OS, I think it is overkill as the default setup. I reckon we would get
> 80-90% of the value, by making our own test image repo, containing minimal
> kernel builds for each machine/target combo we need, together with a tiny
> initrd containing busybox. This could easily be made to boot in 1 second,
> which would make 'make check-acceptance' waaaaay faster, considering how
> many times we boot a guest. This would also solve our problem that we're
> pointing to URLs to download these giant images, and they're frequently
> break URLs.

It's been discussed before but previously the worry has been the hassle
of maintaining such images along with such tediousness as ensuring GPL
compliance. We've outsourced that problem to the upstream.

That said we've got test jobs that run from our QEMU advent calendars
and I added some for Xen testing from a stable Linaro file server
before.

> If we want the re-assurance of running a full guest OS, we could wire
> that up 'make check-acceptance FULL_OS=1' and then set that up as a
> nightly CI job to run post-merge as a sanity-check, where speed does
> not matter
>
>
> Regards,
> Daniel


-- 
Alex Bennée