Re: [Qemu-devel] Who is running QEMU automated tests, and when?

[Michael Clark]

On Fri, Apr 27, 2018 at 2:04 AM, Peter Maydell <address@hidden>
wrote:

> On 26 April 2018 at 14:57, Eduardo Habkost <address@hidden> wrote:
> > Peter, do you have additional tests you run before merging a pull
> > request?  Additional test sets run before tagging a release?
>
> I run make && make check, on a variety of hosts, before merging
> any pull request. That's the only testing I do, and I don't
> do anything more before I do a tag.
>

This is what I do on riscv, however it is not yet automated:

- QEMU upstream `make check` tests are run in Travis and they should all
pass.
- The RISC-V ISA compatibility tests from the `riscv-tests` repository are
  run and currently all of the tests pass.
- Fedora Linux and Debian Linux are used for running self-hosted GCC
bootstraps
  using the latest Linux kernel and GLIBC user-space in the `virt` machine.
  A parallel GCC bootstrap is used for stress testing MTTCG, SMP and the
  interrupt controllers in a multi-core RISC-V virtual machine.
- Linux 4.6.2 with bbl and busybox are used for backward compatibility
testing
  in the `spike_v1.9.1` machine.
- Linux 4.15.0 with bbl and busybox are used for forward compatibility
testing
  in the `spike_v1.10` machine.
- HiFive1 binaries from the `freedom-u-sdk` are used for testing the
  `sifive_e` machine.

I only run the 24-hour type tests if I am changing anything related to
interrupts. We are working on interrupt controller emulation and the SMP
PLIC used by the 'virt' machine was authored by me, however Stefan O'Rear
contributed to the interrupt handling code and I've been very careful to
test that I haven't broken and to fully understand the logic and atomicity
of interrupts. We are working on a new interrupt controller and are
specifically looking at the possibility of adding a GPIO mock that could be
combined with some sort of replay mechanism to test the interrupt
controllers.

We could bring a snapshot of riscv-tests into the QEMU tree however it
would be best if they retained the BSD license so we can share locally
added tests with upstream as upstream is adding tests periodically.

We have recently made changes to the riscv page walker where we have
documented it in such a way that each clause represents a descriprition for
a test case. Our goal is that we would like the code to read similarly to
the RISC-V Privileged ISA specification where possible.

-
https://github.com/riscv/riscv-qemu/blob/qemu-2.13-for-upstream/target/riscv/cpu_helper.c#L210-L243

BTW Andrew Waterman reviewed my update to the page walker and caught a bug.
Linux testing doesn't exercise every dimension of the page permission
checks. We effectively need malware type tests. Multi-core PTE update
atomicity tests will also be interesting. The SiFive verification team have
been working on PMP tests however I am not sure whether these will be made
open source. The RISC-V QEMU PMP code needs testing. It also needs to
implement recent changes to the specification for coarse grain PMPs (4K
granularity) given we use the TLB page size so that PMP can be implemented
with reasonable performance.

We would like to automate all of this and we are working towards that goal.
One of the dependencies for docker tests are the riscv toolchain packages
and we are working on this...

Adding port specific tests, like riscv tests to QEMU not only benefits the
particular port but also exercises a log of generic code such as the TCG,
the IO system, IO mutex, Virt IO and many other shared code paths.