[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Qemu-devel] [ANNOUNCE] qemu-test: a set of tests scripts for QEMU

From: Dor Laor
Subject: Re: [Qemu-devel] [ANNOUNCE] qemu-test: a set of tests scripts for QEMU
Date: Tue, 27 Dec 2011 01:00:24 +0200
User-agent: Mozilla/5.0 (X11; Linux x86_64; rv:9.0) Gecko/20111222 Thunderbird/9.0

On 12/26/2011 05:12 PM, Anthony Liguori wrote:
Hi Dor,

Merry Christmas Anthony,

On 12/25/2011 09:19 AM, Dor Laor wrote:
On 12/19/2011 07:13 PM, Anthony Liguori wrote:

Well, I'm still not convinced that a new standalone package should
handle these
cases instead of kvm autotest. I'll be happy to integrate the tests to
autotest anyway and the more the merrier but imho it's a duplicate.

I'm sure kvm autotest could be taught to do exactly what qemu-test is
doing. But why does kvm autotest have to do everything? I doubt there
would be much code reuse.

I think it's not a bad thing to have multiple test suites when there
isn't considerable overlap.

I agree but in this case, it loos to me that qemu-test is likely to do a subset of what kvm autotest do in a potentially less elegant way but due to the influence of a key maintainer may gain traction and dilute kvm autotest..

It has the following characteristics:

1) It builds a custom kernel and initramfs based on busybox. This is
fairly important to ensure that we can run tests with no device

This can be done easily w/ autotest too.

Okay, please demonstrate :-) The procedure to do this with qemu test is:

$ git clone git://git.qemu.org/qemu-test.git
$ cd qemu-test
$ git submodule update --init
$ make

How would one do this with kvm autotest?

git clone git://github.com/autotest/autotest.git
cd autotest/client/tests/kvm
sudo ./get_started.py

One command shorter and the above downloads F15 automatically and runs it.

(from https://github.com/autotest/autotest/wiki/KVMAutotest-RunGetStartedScript)

2) Tests are scripts that are launched in the initramfs

3) The test controls exactly how QEMU is launched which allows easy
testing of various QEMU options

afaik kvm autotest do the same. It's true that kvm autotest might look
friendly but its may only be this way since it has lots of options.

Actually, kvm-autotest has various layers of abstraction in how QEMU
ends up being launched. As you mention below, those layers are there to
allow for things like using libvirt.

It goes beyond that, since it also related to the monitor interface as well.

That's desirable when you're doing "virt testing", but not so desirably
when you're trying to write specific unit tests against QEMU.

True, one may not need it at all but it's nice that a test for migration/stress/hotplug will be tested directly w/ qemu and libvirt w/ the same effort.

5) The tests execute very quickly, can be run stand alone, and do not
require root privileges

ditto for kvm auotest. It's possible to configure it w/o root too
which is not a
huge issue.

When I say, "run quickly", I mean, they execute very quickly.

/me too

$ time ./qemu-test ~/build/qemu/x86_64-softmmu/qemu-system-x86_64

real 0m4.385s
user 0m1.460s
sys 0m1.860s

That's impressive but it's more of a function of the guest being used - if instead of running a full Fedora install, you'll chose your busybox image w/ -kernel/initrd you'll get a similar result.

I've used kvm-autotest a lot, there is no test in kvm-autotest that is
even close to completing in 4 seconds start to finish. I don't think
kvm-autotest can even run without installing a guest first which puts
puts a simple test more at the 30 minute mark. If we're talking about
TCG testing, then we're in hours territory.

Autotest today may be too verbose by default and will call
echo 3 > /proc/sys/vm/drop_caches but it does not have to be this way and Lucas can easily create a slim mode that does not even need to be root.

If there's a way to run kvm-autotest as a non-privileged user, I've not
figured it out yet.

Again, there is no magic in a test suite so if required we can wave it (maybe we need to do that regardless)

Of course, kvm-autotest can test a lot more things than qemu-test can
since the tests are guest agnostic. That is the primary architectural
difference. By writing the tests to one specific guest (busybox initrd),
they can be extremely fast in execution, but the tests are less general

That makes qemu-test very useful for development testing (such as long
automated bisect runs) but less useful for acceptance testing.

Autotest too can define a family of 'sanity' or 'rapid-bisec-mode' and the tests will be slimmer or each test will have such a mode.

Please compare your own virtio-serial test w/ the autotest version of it:

This single file tests functionality, limits, console, live migration and
performance. Of course one can add a very basic 'hello world' sanity
test too
that will run quickly and will identify basic breakage fast.

Note that virtio_console.py is 2175 LOC whereas virtio-serial.sh is 52.

I bet that virtio_console_super_slim_and_equal_to_qemu_unittest_serial.py will be less than 52 lines. It depends what you like to test. as noted, the above test fully covers all aspects of virtio-serial. It does not have to be this way and I do agree it is nicer to split it to multiple sub files.

There is a lot of value in being able to write simple tests in 50 lines
of code. In fact, the largest qemu-test at the moment is only ~70 lines
of code. And besides finding regressions in my own code (which is the
primary use atm), I've found and fixed a few real bugs in upstream QEMU.

Now imagine what virtio_console.py buys you :)

For a couple hundred lines of bash script, how can it be anything but a
good thing :-)

Noways we abstract kvm autotest so libvirt will be optionally tested
too w/ the
same tests.

Again, I'm not advocating that kvm autotest is a solution for anything
but for
plain guest-host communication, monitor commands, etc it's a really
good tool.

I agree that kvm autotest may be less friendly for developer users
since it
carries allot of options for testing a huge matrix. Lucas and Cleber
are working
these days to add a make kvmautotest target to qemu so one would be
able to
quickly execute autotest and we can think of additional parameters like
sanity-set, migration-set, etc.

We (the QEMU project) need to get more serious about testing. We need to


(quickly) get to a point where we can mandate that features come with
test cases. It is extremely hard to implement touch everything features
with no good way to test all of the different things we support.

kvm autotest cannot fill that role because it's too complicated and too
many layers removed from QEMU.

virtio-console/virtio-serial-bus in about 1400 LOC in QEMU. I'm not
about to mandate that someone writes 2.2k LOC in a python test framework
in order to get 1.4k of code merged in QEMU.

But 50 lines of bash seems like a more than reasonable requirement.

Hmm, what's the coverage of these 50 LOC?
Maybe when your maintainer hat is on, it is indeed a huge task to keep all of qemu platforms/architectures to compile and run so that's enough but when I wear my Red Hat manager role, I wish that it will not only compile and boot but will test as much functionality as possible and will be easily re-used by R&D, QA, libvirt, while all supported guests are _potentially_ exercised, not just busybox.

Since kvm autotest will be there anyway, I wish that developers will contribute 50 LOC to autotest (and not 2.2kloc) and hope the ROI will justify it.

I agree autotest is not perfect but it likes to be such.
If you wish, you can challenge Lucas and Cleber w/ these type of requirements and we'll all improve as a result.



Anthony Liguori

6) They are random by nature with the ability to fix the seed in order
to be used in git-bisect.

I think Gerd had been looking at doing something similar with a custom

I've tried to consider other architectures and had hoped that we could
commit the vmlinuz and initramfs into git so that it was easy to test
other architectures without having a full build environment.
Unfortunately, busybox doesn't link statically with glibc and I can't
see an obvious way to commit binaries while respecting the GPL since we
need to pull glibc into the initramfs.

I know buildroot exists specifically to deal with this but in my
experience, buildroot is very unreliable and extremely heavy weight
since it rebuilds gcc multiple times in order to bootstrap a ulibc

Anyway, the code is available at:


See the README for instructions on how to use it.


Anthony Liguori

reply via email to

[Prev in Thread] Current Thread [Next in Thread]