Re: [Qemu-devel] State of QEMU CI as we enter 4.0

[Cleber Rosa]

On Thu, Mar 14, 2019 at 03:57:06PM +0000, Alex Bennée wrote:
> 
> Hi,
> 
> As we approach stabilisation for 4.0 I thought it would be worth doing a
> review of the current state of CI and stimulate some discussion of where
> it is working for us and what could be improved.
> 
> Testing in Build System
> =======================
> 
> Things seem to be progressing well in this respect. More and more tests
> have been added into the main source tree and they are only a make
> invocation away. These include:
> 
>   check          (includes unit, qapi-schema, qtest and decodetree)
>   check-tcg      (now with system mode tests!)
>   check-softfloat
>   check-block
>   check-acceptance
> 
> Personally check-acceptance is the area I've looked at the least but
> this seems to be the best place for "full life cycle" tests like booting
> kernels and running stress and performance tests. I'm still a little
> unsure how we deal with prebuilt kernels and images here though. Are
> they basically provided by 3rd parties from their websites? Do we mirror
> any of the artefacts we use for these tests?

While it's possible to add any sort of files alongside the tests, and
"get it"[1] from the test[2], this is certainly not desirable for
kernels and other similarly large blobs.  The current approach is to
use well known URLs[3] and download[4][5] those at test run time.

Those are cached locally, automatically on the first run and reused on
subsequent executions.  The caching is helpful for development
environments, but is usually irrelevant to CI environments, where
you'd most often than not get a new machine (or a clean environment).

For now I would, also for the sake of simplicity, keep relying on 3rd
party websites until they prove to be unreliable.  This adds
trasnparency and reproducibility well beyond can be achieved if we
attempt to mirror them to a QEMU sponsored/official location IMO.

> 
> One area of concern is how well this all sits with KVM (and other HW
> accelerators) and how that gets tested. With my ARM hat on I don't
> really see any integration between testing kernel and QEMU changes
> together to catch any problems as the core OS support for KVM gets
> updated.
>

In short, I don't think there should be at the QEMU CI be any
integration testing that changes both KVM and QEMU at once.

But, that's me assuming that the vast majority of changes in QEMU and
KVM can be developed, an tested separately of each other.  That's in
sharp contrast with the the days in which KVM Autotest would build
both the kernel and userspace as part of all test jobs, because of
very frequent dependencies among them.

I'd love to get feedback on this from KVM (and other HW accelerator)
folks.

> Another area I would like to improve is how we expand testing with
> existing test suites. I'm thinking things like LTP and kvm-unit-tests
> which can exercise a bunch of QEMU code but are maybe a bit to big to be
> included in the source tree. Although given we included TestFloat (via a
> git submodule) maybe we shouldn't dismiss that approach? Or is this
> something that could be done via Avocado?
>

Well, there's this:

  https://github.com/avocado-framework-tests/avocado-misc-tests

Which contains close to 300 tests, most of them wrappers for other
test suites, including LTP:

  
https://github.com/avocado-framework-tests/avocado-misc-tests/blob/master/generic/ltp.py

I'm claiming it's the perfect fit for your idea, but sounds like a
good starting point.

> Generally though I think we are doing pretty well at increasing our test
> coverage while making the tests more directly available to developers
> without having to rely on someones personal collection of random
> binaries.
>

+1.

> I wanted to know if we should encode this somewhere in our developer
> documentation:
> 
>   There is a strong preference for new QEMU tests to be integrated with
>   the build system. Developers should be able to (build and) run the new
>   tests locally directly from make.
> 
> ?
>

There should definitely be, if reasonable, a similar experience for
running different types of tests.  Right now, the build system (make
targets) is clearly the common place, so +1.

- Cleber.

[1] - 
https://avocado-framework.readthedocs.io/en/69.0/api/core/avocado.core.html#avocado.core.test.TestData.get_data
[2] - 
https://avocado-framework.readthedocs.io/en/69.0/WritingTests.html#accessing-test-data-files
[3] - 
https://github.com/clebergnu/qemu/blob/sent/target_arch_v5/tests/acceptance/boot_linux_console.py#L68
[4] - 
https://github.com/clebergnu/qemu/blob/sent/target_arch_v5/tests/acceptance/boot_linux_console.py#L92
[5] - 
https://avocado-framework.readthedocs.io/en/69.0/WritingTests.html#fetching-asset-files