[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Qemu-devel] [ANNOUNCE] qemu-test: a set of tests scripts for QEMU

From: Anthony Liguori
Subject: Re: [Qemu-devel] [ANNOUNCE] qemu-test: a set of tests scripts for QEMU
Date: Tue, 27 Dec 2011 09:22:50 -0600
User-agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv: Gecko/20110922 Lightning/1.0b2 Thunderbird/3.1.15

On 12/26/2011 05:00 PM, Dor Laor wrote:
On 12/26/2011 05:12 PM, Anthony Liguori wrote:
Hi Dor,

Merry Christmas Anthony,

Happy Hanukkah!

I think it's not a bad thing to have multiple test suites when there
isn't considerable overlap.

I agree but in this case, it loos to me that qemu-test is likely to do a subset
of what kvm autotest do in a potentially less elegant way but due to the
influence of a key maintainer may gain traction and dilute kvm autotest..

I'm worried that if we take an attitude of doing what we've always done, we'll get the same results we always have.

kvm-autotest is a great test suite, but I'm not happy about the number of regressions that were present in 1.0 even after a month of release candidates. We need to write a lot more tests than we are right now that cover a much broader set of functionality.

It has the following characteristics:

1) It builds a custom kernel and initramfs based on busybox. This is
fairly important to ensure that we can run tests with no device

This can be done easily w/ autotest too.

Okay, please demonstrate :-) The procedure to do this with qemu test is:

$ git clone git://git.qemu.org/qemu-test.git
$ cd qemu-test
$ git submodule update --init
$ make

How would one do this with kvm autotest?

git clone git://github.com/autotest/autotest.git
cd autotest/client/tests/kvm
sudo ./get_started.py

One command shorter and the above downloads F15 automatically and runs it.

(from https://github.com/autotest/autotest/wiki/KVMAutotest-RunGetStartedScript)

I know how to install kvm-autotest :-) You claimed that autotest could easily "build a custom kernel and initramfs based on busybox." I've studied kvm-autotest extensively to determine if it could do what I want it to do and came to the conclusion that it couldn't without significant effort.

The infrastructure assumes that you have a full OS available in the guest. The tests are written in Python and make a variety of assumptions. To my knowledge, it's not very practical to build a busybox environment with Python embedded in it.

This means you need to install a distro and have a disk based install. Not requiring this was one of my primary requirements in creating qemu-test mainly since the resulting images are very large and installing a guest with TCG is extremely slow.

OTOH, you can build a kernel and initrd with a cross compiler and booting a kernel, while not fast under TCG, is not all that slow.

5) The tests execute very quickly, can be run stand alone, and do not
require root privileges

ditto for kvm auotest. It's possible to configure it w/o root too
which is not a
huge issue.

When I say, "run quickly", I mean, they execute very quickly.

/me too

$ time ./qemu-test ~/build/qemu/x86_64-softmmu/qemu-system-x86_64

real 0m4.385s
user 0m1.460s
sys 0m1.860s

That's impressive but it's more of a function of the guest being used - if
instead of running a full Fedora install, you'll chose your busybox image w/
-kernel/initrd you'll get a similar result.

I'm fairly certain that this is not easy to do with kvm-autotest. I'm happy to be proven wrong but I didn't just create qemu-test in a vacuum.

I've used kvm-autotest a lot, there is no test in kvm-autotest that is
even close to completing in 4 seconds start to finish. I don't think
kvm-autotest can even run without installing a guest first which puts
puts a simple test more at the 30 minute mark. If we're talking about
TCG testing, then we're in hours territory.

Autotest today may be too verbose by default and will call
echo 3 > /proc/sys/vm/drop_caches but it does not have to be this way and Lucas
can easily create a slim mode that does not even need to be root.

If it's easy, then by all means, please demonstrate. If qemu-test only serves to show what my requirements for kvm-autotest are and Lucas can implement it all very easily, then I would be happy to discontinue qemu-test immediately.

But as I mentioned above, I don't think it's that simple. You basically can't have python-based guest tests without having a full guest distribution. That means in the very least, every kvm-autotest test will need to be rewritten.

If there's a way to run kvm-autotest as a non-privileged user, I've not
figured it out yet.

Again, there is no magic in a test suite so if required we can wave it (maybe we
need to do that regardless)

I cannot stress enough how important this is. The entire process from git fetch to running the test should not require root privileges. If you are targeting developers, you can't assume that the box is dedicated to testing as you would if you're doing QE testing. That means you need to co-exist with the existing environment and not muck around with network setting and stuff like that.

FWIW, I gave this feedback during Lucas' talk at KVM Forum 2011 so this shouldn't be new news to anyone.

Of course, kvm-autotest can test a lot more things than qemu-test can
since the tests are guest agnostic. That is the primary architectural
difference. By writing the tests to one specific guest (busybox initrd),
they can be extremely fast in execution, but the tests are less general

That makes qemu-test very useful for development testing (such as long
automated bisect runs) but less useful for acceptance testing.

Autotest too can define a family of 'sanity' or 'rapid-bisec-mode' and the tests
will be slimmer or each test will have such a mode.

I've requested a simple config for autotest in the past and even offered to include it in qemu.git as a standard profile.

The simple config that I use for regression testing takes about two hours to complete. I'd prefer to have something that took a few minutes to complete...

Please compare your own virtio-serial test w/ the autotest version of it:

This single file tests functionality, limits, console, live migration and
performance. Of course one can add a very basic 'hello world' sanity
test too
that will run quickly and will identify basic breakage fast.

Note that virtio_console.py is 2175 LOC whereas virtio-serial.sh is 52.

I bet that virtio_console_super_slim_and_equal_to_qemu_unittest_serial.py will
be less than 52 lines.

Okay, then please do demonstrate. I don't mean to be coy here but as I mentioned earlier, I've studied kvm-autotest and based on my understanding of it, I don't think it's possible.

I'd be thrilled to be proven wrong though.

It depends what you like to test. as noted, the above
test fully covers all aspects of virtio-serial. It does not have to be this way
and I do agree it is nicer to split it to multiple sub files.

Looking closely, I think the main difference is that the kvm-autotest test also tests guest semantics. It tests the behavior of module unload, SIGIO, poll, etc.

While those tests are important for distro QE testing, they are not useful for QEMU testing as there's almost nothing that can change in QEMU that's going to affect that behavior. Those are really Linux kernel tests, not QEMU tests.

There is a lot of value in being able to write simple tests in 50 lines
of code. In fact, the largest qemu-test at the moment is only ~70 lines
of code. And besides finding regressions in my own code (which is the
primary use atm), I've found and fixed a few real bugs in upstream QEMU.

Now imagine what virtio_console.py buys you :)

Actually, the only thing that it tests that's a QEMU function is having multiple ports per PCI device. That's something I could add to the qemu-test test with a couple dozen lines of code I suspect.

(quickly) get to a point where we can mandate that features come with
test cases. It is extremely hard to implement touch everything features
with no good way to test all of the different things we support.

kvm autotest cannot fill that role because it's too complicated and too
many layers removed from QEMU.

virtio-console/virtio-serial-bus in about 1400 LOC in QEMU. I'm not
about to mandate that someone writes 2.2k LOC in a python test framework
in order to get 1.4k of code merged in QEMU.

But 50 lines of bash seems like a more than reasonable requirement.

Hmm, what's the coverage of these 50 LOC?

It tests device creation (including via alias), device fingerprinting (although that's a different test), and basic I/O functionality.

Maybe when your maintainer hat is on, it is indeed a huge task to keep all of
qemu platforms/architectures to compile and run so that's enough but when I wear
my Red Hat manager role, I wish that it will not only compile and boot but will
test as much functionality as possible and will be easily re-used by R&D, QA,
libvirt, while all supported guests are _potentially_ exercised, not just 

Right, there are two different use cases here:

1) light, QEMU specific testing, that can be used as a QEMU regression suite. Essentially, the problem to solve is that when doing changes like the memory API, just trying to figure out how to boot a cris guest to confirm that it didn't fall apart is surprisingly time consuming. This is the kind of test suite that a developer would run more or less after every local commit.

2) Thorough integration tests that test the whole stack including the kernel, libvirt, QEMU, and guest software. This is the sort of suite that a distribution would run as part of their QE procedures.

I think these goals are different enough that they deserve to be separate projects. I don't think kvm-autotest would be improved by also taking on (1) as it would force lots of change for very little benefit to (2).

Since kvm autotest will be there anyway, I wish that developers will contribute
50 LOC to autotest (and not 2.2kloc) and hope the ROI will justify it.

I agree autotest is not perfect but it likes to be such.
If you wish, you can challenge Lucas and Cleber w/ these type of requirements
and we'll all improve as a result.

Well consider qemu-test the challenge. It's an existence proof that we can have a very easy to use framework for testing that runs extremely fast and is very easy to write tests for.

I don't think qemu-test would ever be appropriate for distro QE testing as kvm-autotest does an excellent job of this. I really think that this are two fundamentally different use-cases that require specialized tools. Of course, I'm happy to be proven wrong.


Anthony Liguori



Anthony Liguori

6) They are random by nature with the ability to fix the seed in order
to be used in git-bisect.

I think Gerd had been looking at doing something similar with a custom

I've tried to consider other architectures and had hoped that we could
commit the vmlinuz and initramfs into git so that it was easy to test
other architectures without having a full build environment.
Unfortunately, busybox doesn't link statically with glibc and I can't
see an obvious way to commit binaries while respecting the GPL since we
need to pull glibc into the initramfs.

I know buildroot exists specifically to deal with this but in my
experience, buildroot is very unreliable and extremely heavy weight
since it rebuilds gcc multiple times in order to bootstrap a ulibc

Anyway, the code is available at:


See the README for instructions on how to use it.


Anthony Liguori

reply via email to

[Prev in Thread] Current Thread [Next in Thread]