[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Qemu-devel] [ANNOUNCE] qemu-test: a set of tests scripts for QEMU

From: Anthony Liguori
Subject: Re: [Qemu-devel] [ANNOUNCE] qemu-test: a set of tests scripts for QEMU
Date: Mon, 26 Dec 2011 09:12:00 -0600
User-agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv: Gecko/20110922 Lightning/1.0b2 Thunderbird/3.1.15

Hi Dor,

On 12/25/2011 09:19 AM, Dor Laor wrote:
On 12/19/2011 07:13 PM, Anthony Liguori wrote:

Well, I'm still not convinced that a new standalone package should handle these
cases instead of kvm autotest. I'll be happy to integrate the tests to kvm
autotest anyway and the more the merrier but imho it's a duplicate.

I'm sure kvm autotest could be taught to do exactly what qemu-test is doing. But why does kvm autotest have to do everything? I doubt there would be much code reuse.

I think it's not a bad thing to have multiple test suites when there isn't considerable overlap.

It has the following characteristics:

1) It builds a custom kernel and initramfs based on busybox. This is
fairly important to ensure that we can run tests with no device

This can be done easily w/ autotest too.

Okay, please demonstrate :-)  The procedure to do this with qemu test is:

$ git clone git://git.qemu.org/qemu-test.git
$ cd qemu-test
$ git submodule update --init
$ make

How would one do this with kvm autotest?

2) Tests are scripts that are launched in the initramfs

3) The test controls exactly how QEMU is launched which allows easy
testing of various QEMU options

afaik kvm autotest do the same. It's true that kvm autotest might look less
friendly but its may only be this way since it has lots of options.

Actually, kvm-autotest has various layers of abstraction in how QEMU ends up being launched. As you mention below, those layers are there to allow for things like using libvirt.

That's desirable when you're doing "virt testing", but not so desirably when you're trying to write specific unit tests against QEMU.

5) The tests execute very quickly, can be run stand alone, and do not
require root privileges

ditto for kvm auotest. It's possible to configure it w/o root too which is not a
huge issue.

When I say, "run quickly", I mean, they execute very quickly.

$ time ./qemu-test ~/build/qemu/x86_64-softmmu/qemu-system-x86_64 tests/virtio-serial.sh

real    0m4.385s
user    0m1.460s
sys     0m1.860s

I've used kvm-autotest a lot, there is no test in kvm-autotest that is even close to completing in 4 seconds start to finish. I don't think kvm-autotest can even run without installing a guest first which puts puts a simple test more at the 30 minute mark. If we're talking about TCG testing, then we're in hours territory.

If there's a way to run kvm-autotest as a non-privileged user, I've not figured it out yet.

Of course, kvm-autotest can test a lot more things than qemu-test can since the tests are guest agnostic. That is the primary architectural difference. By writing the tests to one specific guest (busybox initrd), they can be extremely fast in execution, but the tests are less general purpose.

That makes qemu-test very useful for development testing (such as long automated bisect runs) but less useful for acceptance testing.

Please compare your own virtio-serial test w/ the autotest version of it:

This single file tests functionality, limits, console, live migration and
performance. Of course one can add a very basic 'hello world' sanity test too
that will run quickly and will identify basic breakage fast.

Note that virtio_console.py is 2175 LOC whereas virtio-serial.sh is 52.

There is a lot of value in being able to write simple tests in 50 lines of code. In fact, the largest qemu-test at the moment is only ~70 lines of code. And besides finding regressions in my own code (which is the primary use atm), I've found and fixed a few real bugs in upstream QEMU.

For a couple hundred lines of bash script, how can it be anything but a good thing :-)

Noways we abstract kvm autotest so libvirt will be optionally tested too w/ the
same tests.

Again, I'm not advocating that kvm autotest is a solution for anything but for
plain guest-host communication, monitor commands, etc it's a really good tool.

I agree that kvm autotest may be less friendly for developer users since it
carries allot of options for testing a huge matrix. Lucas and Cleber are working
these days to add a make kvmautotest target to qemu so one would be able to
quickly execute autotest and we can think of additional parameters like
sanity-set, migration-set, etc.

We (the QEMU project) need to get more serious about testing. We need to (quickly) get to a point where we can mandate that features come with test cases. It is extremely hard to implement touch everything features with no good way to test all of the different things we support.

kvm autotest cannot fill that role because it's too complicated and too many layers removed from QEMU.

virtio-console/virtio-serial-bus in about 1400 LOC in QEMU. I'm not about to mandate that someone writes 2.2k LOC in a python test framework in order to get 1.4k of code merged in QEMU.

But 50 lines of bash seems like a more than reasonable requirement.


Anthony Liguori

6) They are random by nature with the ability to fix the seed in order
to be used in git-bisect.

I think Gerd had been looking at doing something similar with a custom

I've tried to consider other architectures and had hoped that we could
commit the vmlinuz and initramfs into git so that it was easy to test
other architectures without having a full build environment.
Unfortunately, busybox doesn't link statically with glibc and I can't
see an obvious way to commit binaries while respecting the GPL since we
need to pull glibc into the initramfs.

I know buildroot exists specifically to deal with this but in my
experience, buildroot is very unreliable and extremely heavy weight
since it rebuilds gcc multiple times in order to bootstrap a ulibc

Anyway, the code is available at:


See the README for instructions on how to use it.


Anthony Liguori

reply via email to

[Prev in Thread] Current Thread [Next in Thread]