Re: [Qemu-devel] [ANNOUNCE] qemu-test: a set of tests scripts for QEMU

[Avi Kivity]

On 12/28/2011 06:42 PM, Anthony Liguori wrote:
>> In fact using linux as a guest negates that.  First of all, which linux
>> version? if it's fixed, you'll eventually miss functionality and need to
>> migrate.  If it keeps changing, so does your test, and it will keep
>> breaking.
>
>
> The kernel is a git submodule so it's a very specific version.  Yes,
> we may need to bump the version down the road and obviously, if we
> have to change any tests in the process, we can.

Having a full linux source as part of the build process detracts
somewhat from the advantages here.

>
>>
>> Using Linux as a guest allows you to do system tests (ping was an
>> example) but doesn't allow you to do unit tests (test regression where
>> if this bit in that register was set, but if them bits in thar registers
>> were clear, then the packet would be encoded with ROT-26 before being
>> sent out).
>>
>> I think qemu-test needs to use its own drivers which allow full control
>> over what you do with the hardware.  Otherwise, a regression that only
>> shows up in non-Linux guests will not be testable with qemu-test.
>
> I think you're advocating for qtest.  This is another important part
> of my testing strategy.  I haven't received a lot of input on that RFC...
>
> http://mid.gmane.org/address@hidden
>
> But there's certain things that I still consider to be unit testing
> (like basic networking tests) that I don't want to have to write with
> qtest.  I'm not up for writing a TCP/IP stack in Python...

A ping test is not a unit test.

The ping test is already covered by kvm-autotest; just set up a config
that runs just that; after the initial run you'll have a guest installed
so it'll be quick.  If we have a DSL guest it'll even be very quick.

To test the various NIC emulations, you don't need a full TCP stack,
just like you didn't need to write an NTP implementation for qtest's rtc
test.  Instead you just poke values until it sends out a packet.  If you
want to test virtio-net with both direct and indirect buffers, you can
only do that with a unit test, you can't do it using a full linux guest
since it has its own ideas of when to use indirects and when to avoid
them (for example, it may choose to always avoid them).

>> I suggest the following:
>>
>>   - qemu-test: qemu unit tests
>>   - kvm-unit-tests: kvm unit tests
>>   - kvm-autotest: unit test drivers + system tests
>
>
> I would counter with:
>
> - gtester unit tests (test-visitor, test-qobject, etc., qemu-iotest)
> - qtest: low level, single device functional tests
> - kvm-unit-tests: low level, instruction-set level functional tests

Not really.  kvm-unit-tests tests things specific to kvm.

> - qemu-test: higher level functional/coverage tests (multiple device
>              interaction)
> - kvm-autotest: unit/functional test drivers + acceptance testing
>
> Note that the line I'm drawing is acceptance vs. functional testing,
> not unit vs. integration testing.  Technically, our unit tests are
> things like test-visitor.  Everything else is an integration test.
>
> But the separation between kvm-autotest is acceptance testing vs.
> functional testing.
>
> Acceptance testing is, "does Windows boot", "can I create three
> virtio-serial devices".
>
> Obviously, part of acceptance testing is, "does this set of functional
> tests pass".

Seems like a very blurry line.  Especially as the functional test is
weaker than either qtest and kvm-autotest.  I now have to agree with the
others that it's duplicate functionality.  Does it really matter whether
you're creating an image by compiling Linux and assembling and
initramfs, or by downloading Fedora.iso and installing it?  It's testing
exactly the same thing, guest boot and functionality.

Would you add live migration testing to qemu-test?  If yes, you're
duplicating some more.  If not, you're not doing functional or coverage
tests for that functionality.

>
>>> We shouldn't be discussing whether it's possible to merge the two
>>> tools, but rather what the technical benefits of doing so would be.
>>>
>>> Since at this point, there is almost no overlap between the two, I
>>> don't see any actual technical benefit to merging them.  I see benefit
>>> to autotest executing qemu-test, of course.
>>
>> I'd say that running a ping test is a weak version of kvm-autotest's
>> system tests.
>
> Consider the Linux kernel to be our library of functionality to write
> our unit tests.  

I was about to say the same thing, but with a negative implication. 
Using Linux restricts your tests to what Linux does with the devices.

> I don't want to write a TCP/IP stack.  We aren't just grabbing a
> random distro kernel.  We're building one from scratch configured in a
> specific way.

How does that help?

> In theory, we could even host the kernel source on git.qemu.org and
> fork it to add more interesting things in the kernel (although I'd
> prefer not to do this, obviously).

That way lies madness, though it could be argued that it lies all around us.


-- 
error compiling committee.c: too many arguments to function