[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [Qemu-devel] [PATCH] docs: Add docs/devel/testing.rst
From: |
Eric Blake |
Subject: |
Re: [Qemu-devel] [PATCH] docs: Add docs/devel/testing.rst |
Date: |
Mon, 29 Jan 2018 08:51:16 -0600 |
User-agent: |
Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.5.2 |
On 01/28/2018 09:31 PM, Fam Zheng wrote:
> To make our efforts on QEMU testing easier to consume by contributors,
> let's add a document. For example, Patchew reports build errors on
> patches that should be relativly easy to reproduce with a few steps, and
s/relativly/relatively/
> it is much nicer if there is such a documentation that it can refer to.
>
> This focues on how to run existing tests and how to write new test
s/focues/focuses/
> cases, without going into the frameworks themselves.
>
> Signed-off-by: Fam Zheng <address@hidden>
>
> ---
>
I'll try not to repeat comments made by others...
> +This document describes the testing infrastructure in QEMU.
> +
> +"Make check" testings
> +=====================
s/testings/tests/, or 'Testing with "make check"'
> +
> +The "make check" testing family includes most of the C based tests in QEMU.
> For
> +a quick help, run ``make check-help`` from the source tree.
> +
> +The usual way to run these tests is:
> +
> +.. code::
> +
> + make check
> +
> +which includes QAPI schema tests, unit tests, and QTests. Different sub-types
> +of "make check" testings will be explained below.
s/testings/tests/
> +
> +Before running tests, it is best to build QEMU programs first. Some tests
> +expect the executables to exist and will fail with obscure messages if cannot
> +find them.
Should we fix 'make check' to depend on 'make all', so that we don't
have to require this?
> +
> +Unit tests
> +----------
> +
> +Unit tests, which can be invoked with ``make check-unit``, are simple C tests
> +that typically link to individual QEMU objects and exercise them by calling
> +into the modules.
> +
> +If you are writing new code in QEMU, consider adding a unit test, especially
> +for utility modules that are relatively stateless or have few dependencies.
> To
> +add a new unit test:
> +
> +1. Create a new source file. For example, ``tests/test-foo.c``.
> +
> +2. Write the test. Normally you would include the headers file which exports
> + the module API, then verify the interface behaves as expected from your
> + test. The test code should be organized with the glib testing framework.
> + Copy and modify an existing test is usually a good idea.
> +
> +3. Add the test to ``tests/Makefile.include``. First, name the unit test
> + program and add it to ``$(check-unit-y)``; then add a rule to build the
> + executable. Optionally, you can add a magical variable to support
> ``gcov``.
> + For example:
> +
> +.. code::
> +
> + check-unit-y += tests/test-foo$(EXESUF)
> + tests/test-foo$(EXESUF): tests/test-foo.o $(test-util-obj-y)
> + ...
> + gcov-files-test-foo-y = util/foo.c
Is it worth documenting that you can often run 'gdb ./test-foo' after
the fact for a failing test? (Most unit tests don't require any magic
environment variables to be set)
> +
> +QTest
> +-----
> +
> +QTest is a testing framework that simplifies starting QEMU and interacting
> with
> +the virtual machine just like a guest kernel does. It can be very useful to
> +test hardware emulation, for example; it could also control certain aspects
> of
> +QEMU (such as virtual clock stepping), with a specially purposed "qtest"
> +protocol. Refer to the documentation in ``qtest.c`` file for more details of
> +the protocol.
> +
> +QTest cases can be executed with
> +
> +.. code::
> +
> + make check-qtest
> +
> +The QTest library is implemented by ``tests/libqtest.c`` and the API is
> defined
> +in ``tests/libqtest.h``.
> +
> +Consider adding a new QTest case when you are introducing a new virtual
> +hardware, or extending one if you are adding functionalities to an existing
> +virtual device.
> +
> +On top of libqtest, a higher level library, ``libqos``, was created to
> +encapsulate common tasks of device drivers, such as memory management and
> +communicating with system buses or devices. Many virtual device tests use
> +libqos instead of directly calling into libqos.
> +
> +Steps to add a new QTest case are:
> +
> +1. Create a new source file for the test. (More than one file can be added as
> + necessary.) For example, ``tests/test-foo-device.c``. 2. Write the test
> + code with the glib and libqtest/libqos API. See also existing tests and
> the
> + library headers for reference.
> +
> +3. Register the new test in ``tests/Makefile.include``. Add the test
> executable
> + name to an appropriate ``check-qtest-*-y`` variable. For example:
> +
> + ``check-qtest-generic-y = tests/test-foo-device$(EXESUF)``
> +
> +4. Add object dependencies of the executable in the Makefile, including the
> + test source file(s) and other interesting objects. For example:
> +
> + ``tests/test-foo-device$(EXESUF): tests/test-foo-device.o
> $(libqos-obj-y)``
Is it worth documenting that 'make check-qtest' runs EVERY test unless
you define magic environment variables; and if it fails, running 'make
check-qtest V=1' can help identify which test failed? Here, running a
mere 'gdb foo-device-test' is prone to fail (because it is missing magic
environment variables); it's possible to run individual tests under gdb
but it requires more effort (among other things, the V=1 helps you learn
what environment variables to set).
> +
> +Docker based testing
> +====================
'Docker based tests' or 'Testing with docker'
> +
> +Introduction
> +------------
> +
> +Docker testing framework in QEMU utilizes the public Docker images to build
> and
s/^Docker/The Docker/
s/the public/public/
> +test QEMU in predefined and widely accessible Linux environments. This makes
> +it possible to expand the test coverage across distros, toolchain flavors and
> +library versions.
> +
--
Eric Blake, Principal Software Engineer
Red Hat, Inc. +1-919-301-3266
Virtualization: qemu.org | libvirt.org
signature.asc
Description: OpenPGP digital signature
- Re: [Qemu-devel] [PATCH] docs: Add docs/devel/testing.rst, (continued)
Re: [Qemu-devel] [PATCH] docs: Add docs/devel/testing.rst, Alex Bennée, 2018/01/29
Re: [Qemu-devel] [PATCH] docs: Add docs/devel/testing.rst, Stefan Hajnoczi, 2018/01/29
Re: [Qemu-devel] [PATCH] docs: Add docs/devel/testing.rst, Fam Zheng, 2018/01/30
Re: [Qemu-devel] [PATCH] docs: Add docs/devel/testing.rst,
Eric Blake <=
Re: [Qemu-devel] [PATCH] docs: Add docs/devel/testing.rst, no-reply, 2018/01/29