Re: [Qemu-devel] [PATCH 0/5] Acceptance/functional tests

[Fam Zheng]

On Thu, 05/24 20:58, Cleber Rosa wrote:
> TL;DR
> =====
> 
> Another version, with a minimalist approach, to the acceptance tests
> infrastructure for QEMU, based on the Avocado Testing Framework.
> 
> Background
> ==========
> 
> The previous version, still considered an RFC, was sent to the list by
> Eduardo[1] was based on the work held in Amador's branch[2].  After
> reviewing in under a different light, including the experiences
> done and reported by Philippe[3].
> 
> Differences from previous versions
> ==================================
> 
> The main difference is that this series include only the minimal
> changes deemed necessary to have a starting point.  I like to think
> that it's better connected to the QEMU community and project needs,
> and will hopefully allow for a more organic growth.
> 
> Since this version has less features than the previous versions,
> provided it's accepted, these are the next probable development tasks:
> 
>  * Provide a simple variants mechanism to allow the same tests to be
>    run under different targets, machine models and devices (present on
>    the previous versions as a "YAML to Mux" file with architecture
>    definitions)
>  * Implement QEMUMachine migration support (present on the previous
>    version in the "avocado_qemu.test._VM" class)
>  * Implement Guest OS image selection and download (mostly an Avocado
>    feature, paired with a parameter convention and cloud-init support
>    code)
>  * Implement interactive support for Guest OS sessions (present on
>    the previous versions, supported by the aexpect Python module)
> 
> Even though this version shares very little (if any) code with the
> previous versions, the following is a list of noteworthy changes:
> 
>  * Tests directory is now "tests/acceptance" (was "tests/avocado")
>  * Base test class is now "avocado_qemu.Test" (was
>    "avocado_qemu.test.QemuTest")
>  * Base test class is now hosted in "avocado_qemu/__init__.py" (was
>    "avocado_qemu/test.py")
>  * Direct use of "qemu.QEMUMachine", that is, the
>    avocado_qemu.test._VM class is gone
>  * avocado_qemu.Test won't search for QEMU binaries on $PATH.  To use
>    QEMU binary on a custom system location it's necessary to use the
>    "qemu_bin" parameter
>  * Example test in README.rst is distributed as a real test
>    ("test_version.py")
>  * A new "Linux boot console" test, loosely modeled after Phillipe's
>    use case

I like the direction this series is leading us to.  Making it easy to write
'correct' and 'effective' tests is the most important thing IMO, and the few
tests you add here look promising.

I'm also curious to see a test case involving a guest OS image. (I'm also open 
to
a re-implementation of tests/vm/basevm.py with Avocado, if you find it
worthwhile.)

Fam

> 
> Commit summary
> ==============
> 
> Cleber Rosa (5):
>   Add functional/acceptance tests infrastructure
>   scripts/qemu.py: allow adding to the list of extra arguments
>   Acceptance tests: add quick VNC tests
>   scripts/qemu.py: introduce set_console() method
>   Acceptance tests: add Linux kernel boot and console checking test
> 
>  scripts/qemu.py                             | 103 +++++++++++++++-
>  scripts/test_qemu.py                        | 176 +++++++++++++++++++++++++++
>  tests/acceptance/README.rst                 | 141 +++++++++++++++++++++
>  tests/acceptance/avocado_qemu/__init__.py   |  45 +++++++
>  tests/acceptance/test_boot_linux_console.py |  37 ++++++
>  tests/acceptance/test_version.py            |  13 ++
>  tests/acceptance/test_vnc.py                |  50 ++++++++
>  7 files changed, 564 insertions(+), 1 deletion(-)
>  create mode 100644 scripts/test_qemu.py
>  create mode 100644 tests/acceptance/README.rst
>  create mode 100644 tests/acceptance/avocado_qemu/__init__.py
>  create mode 100644 tests/acceptance/test_boot_linux_console.py
>  create mode 100644 tests/acceptance/test_version.py
>  create mode 100644 tests/acceptance/test_vnc.py
> 
> ---
> 
> [1] https://lists.gnu.org/archive/html/qemu-devel/2018-04/msg03443.html
> [2] https://github.com/apahim/qemu/commits/avocado_qemu
> [3] https://lists.gnu.org/archive/html/qemu-devel/2018-04/msg03076.html
>