Re: [Qemu-devel] [PATCH v3] docs: Add docs/devel/testing.rst

[Paolo Bonzini]

On 01/02/2018 03:20, Fam Zheng wrote:
> To make our efforts on QEMU testing easier to consume by contributors,
> let's add a document. For example, Patchew reports build errors on
> patches that should be relatively easy to reproduce with a few steps, and
> it is much nicer if there is such a documentation that it can refer to.
> 
> This focuses on how to run existing tests and how to write new test
> cases, without going into the frameworks themselves.
> 
> The VM based testing section is moved from tests/vm/README which now
> is a single line pointing to the new doc.
> 
> Signed-off-by: Fam Zheng <address@hidden>
> Reviewed-by: Stefan Hajnoczi <address@hidden>

Queued, thanks.

Paolo

> ---
> 
> v3: Fix "development", "reclamation" and "tests". [Eric]
>     Fix "libqtest". [Andrew]
>     Add Stefan's reviewed-by.
> 
> v2: Fix spelling errors and improve wordings. [Stefan, Eric, Thomas]
>     Example test name "test-foo -> foo-test". The mass renaming will be
>     done in another series. [Thomas, Eric]
>     Document how to debug unit tests and qtests. [Eric]
>     Document group permission setup for docker, and the privilege risks.
>     [Alex]
>     Update tests/vm/README.
> ---
>  docs/devel/testing.rst | 486 
> +++++++++++++++++++++++++++++++++++++++++++++++++
>  tests/vm/README        |  90 +--------
>  2 files changed, 487 insertions(+), 89 deletions(-)
>  create mode 100644 docs/devel/testing.rst
> 
> diff --git a/docs/devel/testing.rst b/docs/devel/testing.rst
> new file mode 100644
> index 0000000000..0ca1a2d4b5
> --- /dev/null
> +++ b/docs/devel/testing.rst
> @@ -0,0 +1,486 @@
> +===============
> +Testing in QEMU
> +===============
> +
> +This document describes the testing infrastructure in QEMU.
> +
> +Testing with "make check"
> +=========================
> +
> +The "make check" testing family includes most of the C based tests in QEMU. 
> For
> +a quick help, run ``make check-help`` from the source tree.
> +
> +The usual way to run these tests is:
> +
> +.. code::
> +
> +  make check
> +
> +which includes QAPI schema tests, unit tests, and QTests. Different sub-types
> +of "make check" tests will be explained below.
> +
> +Before running tests, it is best to build QEMU programs first. Some tests
> +expect the executables to exist and will fail with obscure messages if they
> +cannot find them.
> +
> +Unit tests
> +----------
> +
> +Unit tests, which can be invoked with ``make check-unit``, are simple C tests
> +that typically link to individual QEMU object files and exercise them by
> +calling exported functions.
> +
> +If you are writing new code in QEMU, consider adding a unit test, especially
> +for utility modules that are relatively stateless or have few dependencies. 
> To
> +add a new unit test:
> +
> +1. Create a new source file. For example, ``tests/foo-test.c``.
> +
> +2. Write the test. Normally you would include the header file which exports
> +   the module API, then verify the interface behaves as expected from your
> +   test. The test code should be organized with the glib testing framework.
> +   Copying and modifying an existing test is usually a good idea.
> +
> +3. Add the test to ``tests/Makefile.include``. First, name the unit test
> +   program and add it to ``$(check-unit-y)``; then add a rule to build the
> +   executable. Optionally, you can add a magical variable to support 
> ``gcov``.
> +   For example:
> +
> +.. code::
> +
> +  check-unit-y += tests/foo-test$(EXESUF)
> +  tests/foo-test$(EXESUF): tests/foo-test.o $(test-util-obj-y)
> +  ...
> +  gcov-files-foo-test-y = util/foo.c
> +
> +Since unit tests don't require environment variables, the simplest way to 
> debug
> +a unit test failure is often directly invoking it or even running it under
> +``gdb``. However there can still be differences in behavior between ``make``
> +invocations and your manual run, due to ``$MALLOC_PERTURB_`` environment
> +variable (which affects memory reclamation and catches invalid pointers 
> better)
> +and gtester options. If necessary, you can run
> +
> +.. code::
> +  make check-unit V=1
> +
> +and copy the actual command line which executes the unit test, then run
> +it from the command line.
> +
> +QTest
> +-----
> +
> +QTest is a device emulation testing framework.  It can be very useful to test
> +device models; it could also control certain aspects of QEMU (such as virtual
> +clock stepping), with a special purpose "qtest" protocol.  Refer to the
> +documentation in ``qtest.c`` for more details of the protocol.
> +
> +QTest cases can be executed with
> +
> +.. code::
> +
> +   make check-qtest
> +
> +The QTest library is implemented by ``tests/libqtest.c`` and the API is 
> defined
> +in ``tests/libqtest.h``.
> +
> +Consider adding a new QTest case when you are introducing a new virtual
> +hardware, or extending one if you are adding functionalities to an existing
> +virtual device.
> +
> +On top of libqtest, a higher level library, ``libqos``, was created to
> +encapsulate common tasks of device drivers, such as memory management and
> +communicating with system buses or devices. Many virtual device tests use
> +libqos instead of directly calling into libqtest.
> +
> +Steps to add a new QTest case are:
> +
> +1. Create a new source file for the test. (More than one file can be added as
> +   necessary.) For example, ``tests/test-foo-device.c``.
> +
> +2. Write the test code with the glib and libqtest/libqos API. See also 
> existing
> +   tests and the library headers for reference.
> +
> +3. Register the new test in ``tests/Makefile.include``. Add the test 
> executable
> +   name to an appropriate ``check-qtest-*-y`` variable. For example:
> +
> +   ``check-qtest-generic-y = tests/test-foo-device$(EXESUF)``
> +
> +4. Add object dependencies of the executable in the Makefile, including the
> +   test source file(s) and other interesting objects. For example:
> +
> +   ``tests/test-foo-device$(EXESUF): tests/test-foo-device.o 
> $(libqos-obj-y)``
> +
> +Debugging a QTest failure is slightly harder than the unit test because the
> +tests look up QEMU program names in the environment variables, such as
> +``QTEST_QEMU_BINARY`` and ``QTEST_QEMU_IMG``, and also because it is not easy
> +to attach gdb to the QEMU process spawned from the test. But manual invoking
> +and using gdb on the test is still simple to do: find out the actual command
> +from the output of
> +
> +.. code::
> +  make check-qtest V=1
> +
> +which you can run manually.
> +
> +QAPI schema tests
> +-----------------
> +
> +The QAPI schema tests validate the QAPI parser used by QMP, by feeding
> +predefined input to the parser and comparing the result with the reference
> +output.
> +
> +The input/output data is managed under the ``tests/qapi-schema`` directory.
> +Each test case includes four files that have a common base name:
> +
> +  * ``${casename}.json`` - the file contains the JSON input for feeding the
> +    parser
> +  * ``${casename}.out`` - the file contains the expected stdout from the 
> parser
> +  * ``${casename}.err`` - the file contains the expected stderr from the 
> parser
> +  * ``${casename}.exit`` - the expected error code
> +
> +Consider adding a new QAPI schema test when you are making a change on the 
> QAPI
> +parser (either fixing a bug or extending/modifying the syntax). To do this:
> +
> +1. Add four files for the new case as explained above. For example:
> +
> +  ``$EDITOR tests/qapi-schema/foo.{json,out,err,exit}``.
> +
> +2. Add the new test in ``tests/Makefile.include``. For example:
> +
> +  ``qapi-schema += foo.json``
> +
> +check-block
> +-----------
> +
> +``make check-block`` is a legacy command to invoke block layer iotests and is
> +rarely used. See "QEMU iotests" section below for more information.
> +
> +GCC gcov support
> +----------------
> +
> +``gcov`` is a GCC tool to analyze the testing coverage by instrumenting the
> +tested code. To use it, configure QEMU with ``--enable-gcov`` option and 
> build.
> +Then run ``make check`` as usual. There will be additional ``gcov`` output as
> +the testing goes on, showing the test coverage percentage numbers per 
> analyzed
> +source file. More detailed reports can be obtained by running ``gcov`` 
> command
> +on the output files under ``$build_dir/tests/``, please read the ``gcov``
> +documentation for more information.
> +
> +QEMU iotests
> +============
> +
> +QEMU iotests, under the directory ``tests/qemu-iotests``, is the testing
> +framework widely used to test block layer related features. It is higher 
> level
> +than "make check" tests and 99% of the code is written in bash or Python
> +scripts.  The testing success criteria is golden output comparison, and the
> +test files are named with numbers.
> +
> +To run iotests, make sure QEMU is built successfully, then switch to the
> +``tests/qemu-iotests`` directory under the build directory, and run 
> ``./check``
> +with desired arguments from there.
> +
> +By default, "raw" format and "file" protocol is used; all tests will be
> +executed, except the unsupported ones. You can override the format and 
> protocol
> +with arguments:
> +
> +.. code::
> +
> +  # test with qcow2 format
> +  ./check -qcow2
> +  # or test a different protocol
> +  ./check -nbd
> +
> +It's also possible to list test numbers explicitly:
> +
> +.. code::
> +
> +  # run selected cases with qcow2 format
> +  ./check -qcow2 001 030 153
> +
> +Cache mode can be selected with the "-c" option, which may help reveal bugs
> +that are specific to certain cache mode.
> +
> +More options are supported by the ``./check`` script, run ``./check -h`` for
> +help.
> +
> +Writing a new test case
> +-----------------------
> +
> +Consider writing a tests case when you are making any changes to the block
> +layer. An iotest case is usually the choice for that. There are already many
> +test cases, so it is possible that extending one of them may achieve the goal
> +and save the boilerplate to create one.  (Unfortunately, there isn't a 100%
> +reliable way to find a related one out of hundreds of tests.  One approach is
> +using ``git grep``.)
> +
> +Usually an iotest case consists of two files. One is an executable that
> +produces output to stdout and stderr, the other is the expected reference
> +output. They are given the same number in file names. E.g. Test script 
> ``055``
> +and reference output ``055.out``.
> +
> +In rare cases, when outputs differ between cache mode ``none`` and others, a
> +``.out.nocache`` file is added. In other cases, when outputs differ between
> +image formats, more than one ``.out`` files are created ending with the
> +respective format names, e.g. ``178.out.qcow2`` and ``178.out.raw``.
> +
> +There isn't a hard rule about how to write a test script, but a new test is
> +usually a (copy and) modification of an existing case.  There are a few
> +commonly used ways to create a test:
> +
> +* A Bash script. It will make use of several environmental variables related
> +  to the testing procedure, and could source a group of ``common.*`` 
> libraries
> +  for some common helper routines.
> +
> +* A Python unittest script. Import ``iotests`` and create a subclass of
> +  ``iotests.QMPTestCase``, then call ``iotests.main`` method. The downside of
> +  this approach is that the output is too scarce, and the script is 
> considered
> +  harder to debug.
> +
> +* A simple Python script without using unittest module. This could also 
> import
> +  ``iotests`` for launching QEMU and utilities etc, but it doesn't inherit
> +  from ``iotests.QMPTestCase`` therefore doesn't use the Python unittest
> +  execution. This is a combination of 1 and 2.
> +
> +Pick the language per your preference since both Bash and Python have
> +comparable library support for invoking and interacting with QEMU programs. 
> If
> +you opt for Python, it is strongly recommended to write Python 3 compatible
> +code.
> +
> +Docker based tests
> +==================
> +
> +Introduction
> +------------
> +
> +The Docker testing framework in QEMU utilizes public Docker images to build 
> and
> +test QEMU in predefined and widely accessible Linux environments.  This makes
> +it possible to expand the test coverage across distros, toolchain flavors and
> +library versions.
> +
> +Prerequisites
> +-------------
> +
> +Install "docker" with the system package manager and start the Docker service
> +on your development machine, then make sure you have the privilege to run
> +Docker commands. Typically it means setting up passwordless ``sudo docker``
> +command or login as root. For example:
> +
> +.. code::
> +
> +  $ sudo yum install docker
> +  $ # or `apt-get install docker` for Ubuntu, etc.
> +  $ sudo systemctl start docker
> +  $ sudo docker ps
> +
> +The last command should print an empty table, to verify the system is ready.
> +
> +An alternative method to set up permissions is by adding the current user to
> +"docker" group and making the docker daemon socket file (by default
> +``/var/run/docker.sock``) accessible to the group:
> +
> +.. code::
> +
> +  $ sudo groupadd docker
> +  $ sudo usermod $USER -G docker
> +  $ sudo chown :docker /var/run/docker.sock
> +
> +Note that any one of above configurations makes it possible for the user to
> +exploit the whole host with Docker bind mounting or other privileged
> +operations.  So only do it on development machines.
> +
> +Quickstart
> +----------
> +
> +From source tree, type ``make docker`` to see the help. Testing can be 
> started
> +without configuring or building QEMU (``configure`` and ``make`` are done in
> +the container, with parameters defined by the make target):
> +
> +.. code::
> +
> +  make address@hidden
> +
> +This will create a container instance using the ``min-glib`` image (the image
> +is downloaded and initialized automatically), in which the ``test-build`` job
> +is executed.
> +
> +Images
> +------
> +
> +Along with many other images, the ``min-glib`` image is defined in a 
> Dockerfile
> +in ``tests/docker/dockefiles/``, called ``min-glib.docker``. ``make docker``
> +command will list all the available images.
> +
> +To add a new image, simply create a new ``.docker`` file under the
> +``tests/docker/dockerfiles/`` directory.
> +
> +A ``.pre`` script can be added beside the ``.docker`` file, which will be
> +executed before building the image under the build context directory. This is
> +mainly used to do necessary host side setup. One such setup is 
> ``binfmt_misc``,
> +for example, to make qemu-user powered cross build containers work.
> +
> +Tests
> +-----
> +
> +Different tests are added to cover various configurations to build and test
> +QEMU.  Docker tests are the executables under ``tests/docker`` named
> +``test-*``. They are typically shell scripts and are built on top of a shell
> +library, ``tests/docker/common.rc``, which provides helpers to find the QEMU
> +source and build it.
> +
> +The full list of tests is printed in the ``make docker`` help.
> +
> +Tools
> +-----
> +
> +There are executables that are created to run in a specific Docker 
> environment.
> +This makes it easy to write scripts that have heavy or special dependencies,
> +but are still very easy to use.
> +
> +Currently the only tool is ``travis``, which mimics the Travis-CI tests in a
> +container. It runs in the ``travis`` image:
> +
> +.. code::
> +
> +  make address@hidden
> +
> +Debugging a Docker test failure
> +-------------------------------
> +
> +When CI tasks, maintainers or yourself report a Docker test failure, follow 
> the
> +below steps to debug it:
> +
> +1. Locally reproduce the failure with the reported command line. E.g. run
> +   ``make address@hidden J=8``.
> +2. Add "V=1" to the command line, try again, to see the verbose output.
> +3. Further add "DEBUG=1" to the command line. This will pause in a shell 
> prompt
> +   in the container right before testing starts. You could either manually
> +   build QEMU and run tests from there, or press Ctrl-D to let the Docker
> +   testing continue.
> +4. If you press Ctrl-D, the same building and testing procedure will begin, 
> and
> +   will hopefully run into the error again. After that, you will be dropped 
> to
> +   the prompt for debug.
> +
> +Options
> +-------
> +
> +Various options can be used to affect how Docker tests are done. The full
> +list is in the ``make docker`` help text. The frequently used ones are:
> +
> +* ``V=1``: the same as in top level ``make``. It will be propagated to the
> +  container and enable verbose output.
> +* ``J=$N``: the number of parallel tasks in make commands in the container,
> +  similar to the ``-j $N`` option in top level ``make``. (The ``-j`` option 
> in
> +  top level ``make`` will not be propagated into the container.)
> +* ``DEBUG=1``: enables debug. See the previous "Debugging a Docker test
> +  failure" section.
> +
> +VM testing
> +==========
> +
> +This test suite contains scripts that bootstrap various guest images that 
> have
> +necessary packages to build QEMU. The basic usage is documented in 
> ``Makefile``
> +help which is displayed with ``make vm-test``.
> +
> +Quickstart
> +----------
> +
> +Run ``make vm-test`` to list available make targets. Invoke a specific make
> +command to run build test in an image. For example, ``make vm-build-freebsd``
> +will build the source tree in the FreeBSD image. The command can be executed
> +from either the source tree or the build dir; if the former, ``./configure`` 
> is
> +not needed. The command will then generate the test image in ``./tests/vm/``
> +under the working directory.
> +
> +Note: images created by the scripts accept a well-known RSA key pair for SSH
> +access, so they SHOULD NOT be exposed to external interfaces if you are
> +concerned about attackers taking control of the guest and potentially
> +exploiting a QEMU security bug to compromise the host.
> +
> +QEMU binary
> +-----------
> +
> +By default, qemu-system-x86_64 is searched in $PATH to run the guest. If 
> there
> +isn't one, or if it is older than 2.10, the test won't work. In this case,
> +provide the QEMU binary in env var: ``QEMU=/path/to/qemu-2.10+``.
> +
> +Make jobs
> +---------
> +
> +The ``-j$X`` option in the make command line is not propagated into the VM,
> +specify ``J=$X`` to control the make jobs in the guest.
> +
> +Debugging
> +---------
> +
> +Add ``DEBUG=1`` and/or ``V=1`` to the make command to allow interactive
> +debugging and verbose output. If this is not enough, see the next section.
> +
> +Manual invocation
> +-----------------
> +
> +Each guest script is an executable script with the same command line options.
> +For example to work with the netbsd guest, use ``$QEMU_SRC/tests/vm/netbsd``:
> +
> +.. code::
> +
> +    $ cd $QEMU_SRC/tests/vm
> +
> +    # To bootstrap the image
> +    $ ./netbsd --build-image --image /var/tmp/netbsd.img
> +    <...>
> +
> +    # To run an arbitrary command in guest (the output will not be echoed 
> unless
> +    # --debug is added)
> +    $ ./netbsd --debug --image /var/tmp/netbsd.img uname -a
> +
> +    # To build QEMU in guest
> +    $ ./netbsd --debug --image /var/tmp/netbsd.img --build-qemu $QEMU_SRC
> +
> +    # To get to an interactive shell
> +    $ ./netbsd --interactive --image /var/tmp/netbsd.img sh
> +
> +Adding new guests
> +-----------------
> +
> +Please look at existing guest scripts for how to add new guests.
> +
> +Most importantly, create a subclass of BaseVM and implement ``build_image()``
> +method and define ``BUILD_SCRIPT``, then finally call ``basevm.main()`` from
> +the script's ``main()``.
> +
> +* Usually in ``build_image()``, a template image is downloaded from a
> +  predefined URL. ``BaseVM._download_with_cache()`` takes care of the cache 
> and
> +  the checksum, so consider using it.
> +
> +* Once the image is downloaded, users, SSH server and QEMU build deps should
> +  be set up:
> +
> +  - Root password set to ``BaseVM.ROOT_PASS``
> +  - User ``BaseVM.GUEST_USER`` is created, and password set to
> +    ``BaseVM.GUEST_PASS``
> +  - SSH service is enabled and started on boot,
> +    ``$QEMU_SRC/tests/keys/id_rsa.pub`` is added to ssh's ``authorized_keys``
> +    file of both root and the normal user
> +  - DHCP client service is enabled and started on boot, so that it can
> +    automatically configure the virtio-net-pci NIC and communicate with QEMU
> +    user net (10.0.2.2)
> +  - Necessary packages are installed to untar the source tarball and build
> +    QEMU
> +
> +* Write a proper ``BUILD_SCRIPT`` template, which should be a shell script 
> that
> +  untars a raw virtio-blk block device, which is the tarball data blob of the
> +  QEMU source tree, then configure/build it. Running "make check" is also
> +  recommended.
> +
> +Image fuzzer testing
> +====================
> +
> +An image fuzzer was added to exercise format drivers. Currently only qcow2 is
> +supported. To start the fuzzer, run
> +
> +.. code::
> +
> +  tests/image-fuzzer/runner.py -c '[["qemu-img", "info", "$test_img"]]' 
> /tmp/test qcow2
> +
> +Alternatively, some command different from "qemu-img info" can be tested, by
> +changing the ``-c`` option.
> diff --git a/tests/vm/README b/tests/vm/README
> index ae53dce6ee..f9c04cc0e7 100644
> --- a/tests/vm/README
> +++ b/tests/vm/README
> @@ -1,89 +1 @@
> -=== VM test suite to run build in guests ===
> -
> -== Intro ==
> -
> -This test suite contains scripts that bootstrap various guest images that 
> have
> -necessary packages to build QEMU. The basic usage is documented in Makefile
> -help which is displayed with "make vm-test".
> -
> -== Quick start ==
> -
> -Run "make vm-test" to list available make targets. Invoke a specific make
> -command to run build test in an image. For example, "make vm-build-freebsd"
> -will build the source tree in the FreeBSD image. The command can be executed
> -from either the source tree or the build dir; if the former, ./configure is 
> not
> -needed. The command will then generate the test image in ./tests/vm/ under 
> the
> -working directory.
> -
> -Note: images created by the scripts accept a well-known RSA key pair for SSH
> -access, so they SHOULD NOT be exposed to external interfaces if you are
> -concerned about attackers taking control of the guest and potentially
> -exploiting a QEMU security bug to compromise the host.
> -
> -== QEMU binary ==
> -
> -By default, qemu-system-x86_64 is searched in $PATH to run the guest. If 
> there
> -isn't one, or if it is older than 2.10, the test won't work. In this case,
> -provide the QEMU binary in env var: QEMU=/path/to/qemu-2.10+.
> -
> -== Make jobs ==
> -
> -The "-j$X" option in the make command line is not propagated into the VM,
> -specify "J=$X" to control the make jobs in the guest.
> -
> -== Debugging ==
> -
> -Add "DEBUG=1" and/or "V=1" to the make command to allow interactive debugging
> -and verbose output. If this is not enough, see the next section.
> -
> -== Manual invocation ==
> -
> -Each guest script is an executable script with the same command line options.
> -For example to work with the netbsd guest, use $QEMU_SRC/tests/vm/netbsd:
> -
> -    $ cd $QEMU_SRC/tests/vm
> -
> -    # To bootstrap the image
> -    $ ./netbsd --build-image --image /var/tmp/netbsd.img
> -    <...>
> -
> -    # To run an arbitrary command in guest (the output will not be echoed 
> unless
> -    # --debug is added)
> -    $ ./netbsd --debug --image /var/tmp/netbsd.img uname -a
> -
> -    # To build QEMU in guest
> -    $ ./netbsd --debug --image /var/tmp/netbsd.img --build-qemu $QEMU_SRC
> -
> -    # To get to an interactive shell
> -    $ ./netbsd --interactive --image /var/tmp/netbsd.img sh
> -
> -== Adding new guests ==
> -
> -Please look at existing guest scripts for how to add new guests.
> -
> -Most importantly, create a subclass of BaseVM and implement build_image()
> -method and define BUILD_SCRIPT, then finally call basevm.main() from the
> -script's main().
> -
> -  - Usually in build_image(), a template image is downloaded from a 
> predefined
> -    URL. BaseVM._download_with_cache() takes care of the cache and the
> -    checksum, so consider using it.
> -
> -  - Once the image is downloaded, users, SSH server and QEMU build deps 
> should
> -    be set up:
> -
> -    * Root password set to BaseVM.ROOT_PASS
> -    * User BaseVM.GUEST_USER is created, and password set to 
> BaseVM.GUEST_PASS
> -    * SSH service is enabled and started on boot,
> -      $QEMU_SRC/tests/keys/id_rsa.pub is added to ssh's "authorized_keys" 
> file
> -      of both root and the normal user
> -    * DHCP client service is enabled and started on boot, so that it can
> -      automatically configure the virtio-net-pci NIC and communicate with 
> QEMU
> -      user net (10.0.2.2)
> -    * Necessary packages are installed to untar the source tarball and build
> -      QEMU
> -
> -  - Write a proper BUILD_SCRIPT template, which should be a shell script that
> -    untars a raw virtio-blk block device, which is the tarball data blob of 
> the
> -    QEMU source tree, then configure/build it. Running "make check" is also
> -    recommended.
> +See docs/devel/testing.rst for help.
>