[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Qemu-devel] [ANNOUNCE] qemu-test: a set of tests scripts for QEMU

From: Dor Laor
Subject: Re: [Qemu-devel] [ANNOUNCE] qemu-test: a set of tests scripts for QEMU
Date: Thu, 29 Dec 2011 16:38:42 +0200
User-agent: Mozilla/5.0 (X11; Linux x86_64; rv:9.0) Gecko/20111222 Thunderbird/9.0

On 12/28/2011 07:21 PM, Avi Kivity wrote:
I think you're advocating for qtest.  This is another important part
>  of my testing strategy.  I haven't received a lot of input on that RFC...
>  http://mid.gmane.org/address@hidden
>  But there's certain things that I still consider to be unit testing
>  (like basic networking tests) that I don't want to have to write with
>  qtest.  I'm not up for writing a TCP/IP stack in Python...
A ping test is not a unit test.

The ping test is already covered by kvm-autotest; just set up a config
that runs just that; after the initial run you'll have a guest installed
so it'll be quick.  If we have a DSL guest it'll even be very quick.

To test the various NIC emulations, you don't need a full TCP stack,
just like you didn't need to write an NTP implementation for qtest's rtc
test.  Instead you just poke values until it sends out a packet.  If you
want to test virtio-net with both direct and indirect buffers, you can
only do that with a unit test, you can't do it using a full linux guest
since it has its own ideas of when to use indirects and when to avoid
them (for example, it may choose to always avoid them).

>>  I suggest the following:
>>     - qemu-test: qemu unit tests
>>     - kvm-unit-tests: kvm unit tests
>>     - kvm-autotest: unit test drivers + system tests
>  I would counter with:
>  - gtester unit tests (test-visitor, test-qobject, etc., qemu-iotest)

+1 for qtester

>  - qtest: low level, single device functional tests

+1 for it

>  - kvm-unit-tests: low level, instruction-set level functional tests
Not really.  kvm-unit-tests tests things specific to kvm.

>  - qemu-test: higher level functional/coverage tests (multiple device
>                interaction)
>  - kvm-autotest: unit/functional test drivers + acceptance testing
>  Note that the line I'm drawing is acceptance vs. functional testing,
>  not unit vs. integration testing.  Technically, our unit tests are
>  things like test-visitor.  Everything else is an integration test.
>  But the separation between kvm-autotest is acceptance testing vs.
>  functional testing.
>  Acceptance testing is, "does Windows boot", "can I create three
>  virtio-serial devices".
>  Obviously, part of acceptance testing is, "does this set of functional
>  tests pass".
Seems like a very blurry line.  Especially as the functional test is
weaker than either qtest and kvm-autotest.  I now have to agree with the
others that it's duplicate functionality.  Does it really matter whether
you're creating an image by compiling Linux and assembling and
initramfs, or by downloading Fedora.iso and installing it?  It's testing
exactly the same thing, guest boot and functionality.

Would you add live migration testing to qemu-test?  If yes, you're
duplicating some more.  If not, you're not doing functional or coverage
tests for that functionality.

From the recent threads it looks to me that the 2 advantages of qemu-test over kvm-autotest are:

1. python is not used within the guest
2. qemu-test is smaller and simpler

Except for the above 2, kvm autotest is superior. My main motivation to merge qemu-test and kvm autotest is that I fear that qemu-test will be the only requirement for committing stuff for qemu and developers will be excused from their 'duty' by writing a 50 LOC shell script and assume they done w/ testing. In addition to that, the 50 LOC will be too basic and only provide value for basic functionality tests and kvm autotest folks will need to rewrite all of the matching content and beyond.

Those above 2 advantages can be solve:

1. python is not used within the guest
   - One option is just to use the same shell code in kvm autotest w/o
     no python code and use the same kernel/initramfs.
   - Another way is to use a plain linux distro w/ python and boot it
     into single user test mode and potentially use a S4 guest snapshot
     or external snapshot to shorten its boot time.
     You'll get faster boot time and friendlier code.

2. qemu-test is smaller and simpler
   kvm autotest will have a fast, out of the box mode especially
   designed to answer your requirements.
   It's mainly a matter of preparing a fastTestMode.py that will setup
all of the above in a similar way that today's https://github.com/autotest/autotest/blob/master/client/tests/kvm/get_started.py

Lastly, each test can have a fast|basic|sanity overloaded function that will be called by this.

>>>  We shouldn't be discussing whether it's possible to merge the two
>>>  tools, but rather what the technical benefits of doing so would be.
>>>  Since at this point, there is almost no overlap between the two, I
>>>  don't see any actual technical benefit to merging them.  I see benefit
>>>  to autotest executing qemu-test, of course.
>>  I'd say that running a ping test is a weak version of kvm-autotest's
>>  system tests.
>  Consider the Linux kernel to be our library of functionality to write
>  our unit tests.
I was about to say the same thing, but with a negative implication.
Using Linux restricts your tests to what Linux does with the devices.

>  I don't want to write a TCP/IP stack.  We aren't just grabbing a
>  random distro kernel.  We're building one from scratch configured in a
>  specific way.
How does that help?

>  In theory, we could even host the kernel source on git.qemu.org and
>  fork it to add more interesting things in the kernel (although I'd
>  prefer not to do this, obviously).
That way lies madness, though it could be argued that it lies all around us.

reply via email to

[Prev in Thread] Current Thread [Next in Thread]