[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Qemu-devel] [ANNOUNCE] qemu-test: a set of tests scripts for QEMU

From: Anthony Liguori
Subject: Re: [Qemu-devel] [ANNOUNCE] qemu-test: a set of tests scripts for QEMU
Date: Wed, 28 Dec 2011 10:42:05 -0600
User-agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv: Gecko/20110922 Lightning/1.0b2 Thunderbird/3.1.15

On 12/28/2011 09:01 AM, Avi Kivity wrote:
On 12/28/2011 04:27 PM, Anthony Liguori wrote:
Maybe I've used the wrong wording. I got the feeling that, besides
testing qemu
the way you need it, keeping qemu-test simple was really important.

Simple is always important.  In the case of qemu-test, there are some
important trade offs made to keep things simple and fast.  It doesn't
try to work with arbitrary guests which means the tests need to handle
only one environment.  The guest is pre-made to have exactly what is
needed for the tests so there is no setup required.  The guest is as
small as possible such that the test can run as quickly as possible.

In fact using linux as a guest negates that.  First of all, which linux
version? if it's fixed, you'll eventually miss functionality and need to
migrate.  If it keeps changing, so does your test, and it will keep

The kernel is a git submodule so it's a very specific version. Yes, we may need to bump the version down the road and obviously, if we have to change any tests in the process, we can.

Using Linux as a guest allows you to do system tests (ping was an
example) but doesn't allow you to do unit tests (test regression where
if this bit in that register was set, but if them bits in thar registers
were clear, then the packet would be encoded with ROT-26 before being
sent out).

I think qemu-test needs to use its own drivers which allow full control
over what you do with the hardware.  Otherwise, a regression that only
shows up in non-Linux guests will not be testable with qemu-test.

I think you're advocating for qtest. This is another important part of my testing strategy. I haven't received a lot of input on that RFC...


But there's certain things that I still consider to be unit testing (like basic networking tests) that I don't want to have to write with qtest. I'm not up for writing a TCP/IP stack in Python...

There is no need to have a single tool that meets every possible
need.  In fact, the Unix tradition is to have separate single purposed

Those single purpose tools, if successful, tend to grow more purposes
(cf. qemu), and if unsuccessful, tend to lose purpose.

Having two test tools it not a bad thing provided that the overlap
isn't significant.

This is important, if the boundary isn't clear, then it will grow more
fuzzy in time.

I suggest the following:

  - qemu-test: qemu unit tests
  - kvm-unit-tests: kvm unit tests
  - kvm-autotest: unit test drivers + system tests

I would counter with:

- gtester unit tests (test-visitor, test-qobject, etc., qemu-iotest)
- qtest: low level, single device functional tests
- kvm-unit-tests: low level, instruction-set level functional tests
- qemu-test: higher level functional/coverage tests (multiple device
- kvm-autotest: unit/functional test drivers + acceptance testing

Note that the line I'm drawing is acceptance vs. functional testing, not unit vs. integration testing. Technically, our unit tests are things like test-visitor. Everything else is an integration test.

But the separation between kvm-autotest is acceptance testing vs. functional testing.

Acceptance testing is, "does Windows boot", "can I create three virtio-serial devices".

Obviously, part of acceptance testing is, "does this set of functional tests 

We shouldn't be discussing whether it's possible to merge the two
tools, but rather what the technical benefits of doing so would be.

Since at this point, there is almost no overlap between the two, I
don't see any actual technical benefit to merging them.  I see benefit
to autotest executing qemu-test, of course.

I'd say that running a ping test is a weak version of kvm-autotest's
system tests.

Consider the Linux kernel to be our library of functionality to write our unit tests. I don't want to write a TCP/IP stack. We aren't just grabbing a random distro kernel. We're building one from scratch configured in a specific way.

In theory, we could even host the kernel source on git.qemu.org and fork it to add more interesting things in the kernel (although I'd prefer not to do this, obviously).

 Running a synthetic test that pokes values into memory
and mmio and sees a packet coming out is a unit test (the latter can in
fact be executed without a guest at all, just a table driving calls to
the memory and irq APIs).

Just putting the code in kvm-autotest.git in a directory doesn't make
sense to me.  Beyond the lack of a technical reason to do so,
logistically, it makes it harder for me to ask people to submit test
cases with a patch series if I can't actually apply the test case when
I'm applying the patches to qemu.git.

If qemu-test didn't use large submodules (like linux.git), I would
have made qemu-test part of qemu.git.  As far as I'm concerned,
qemu-test.git is just an extension to qemu.git.

Why not just put it in qemu.git?

Actually, having thought a bit about it, I think we can move qemu-test into qemu.git and just build the initramfs in a separate repository. That will make it easier to host binary packages of the later on qemu.org.


Anthony Liguori

reply via email to

[Prev in Thread] Current Thread [Next in Thread]