Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests

[Ademar Reis]

On Thu, Mar 08, 2012 at 08:07:27PM -0300, Lucas Meneghel Rodrigues wrote:
> On 03/08/2012 06:24 PM, Anthony Liguori wrote:
> >>
> >>Cons:
> >>- Lot of code will be duplicated to cover the main code paths:
> >>writting tests will require writting/supporting considerable
> >>ammount of code (that already exists in autotest).
> >
> >Again, existence proof that this isn't true.
> 
> Case in point, the virtio test (that uses an auxiliary script to
> send data to the host). Can you tell me if both tests cover even
> remotely the same amount of functionality?
> 
> https://github.com/autotest/autotest/blob/master/client/tests/kvm/tests/virtio_console.py
> 
> https://github.com/autotest/autotest/blob/master/client/virt/scripts/virtio_console_guest.py
> 
> Here is the qemu-test version
> 
> http://git.qemu.org/?p=qemu-test.git;a=blob;f=tests/virtio-serial.sh;h=e95ae6e0b63758262919702d51a9c83bebe2fb08;hb=master
> 
> What the qemu-test version covers:
>  * host starts qemu with one virtio console device backed by a file
>  * guest verifies if the name of the device is correct
>  * guest writes to the console device
>  * host verifies if guest wrote to the virtio console
> 
> What the virtio-console covers:
> 
>  * Sends data between host and guest back and forth, validates the
> data being sent, for both small and large amounts of data, both
> random or sequential.
>  * Tests write/send in blocking, polling, selecting mode, with port
> mode sync/async
>  * Verifies if the maximum amount of ports was created and it's
> available in the guest
>  * Tries lseek on the device
>  * Verifies if concomitant access to a single port passes or fails
>  * Verifies data throughput and resource utilization
>  * Repeats the data transfer with live migration to see if the port
> connections survive
>  * Keeps an eye on the guest OS to see if we have kernel panics, and
> if it does, it'll clean up and put the guest in a working state so
> other functionality can be checked
>  * Probably something else I'm forgetting right now.
> 

Thanks for the example Lucas. :)

BTW, this is a QE-level test. Simpler tests may be sufficient for
developers and for TDD, but, if one implements even a much
simpler test using libautotest, he gets these benefits for free:

1. QE or somebody else may extend the test using the autotest
goodies the original developer was not aware of. Patches from QE
will flow to qemu.git. That's the integration with QE I was
talking about.

2. The test can run in the autotest grid, where a large matrix of
configurations and usage scenarios are used. That's possible
because we don't have much stuff hardcoded.

3. Autotest has a lot of checks for robustness. One of the main
problems of keeping thousands of tests running is to keep them
stable, without false positives.

Lucas: could you ellaborate more on what we get "for free" when
we use libautotest, even when writting a very simple
developer-level test?

> Good luck implementing that in a shell script. I'd love to see how
> you implement that amount of coverage in less lines in shell.
> 
> So, more coverage, more code. It's as simple as that. We don't write
> code just for the sake of it.
> 
> >>- QE will be alienated from the qemu test effort. There will be
> >>no integration between the QE efforts and the maintenance of
> >>the qemu developer-level tests.
> >
> >I think we're a pretty friendly and open community :-) There is no
> >reason that QE should be "alienated" unless folks are choosing not to
> >participate upstream.
> >
> >>- You don't get the goodies from autotest (standardized system
> >>info collection, video recording, grid support, etc).
> >>- The new tests will work only on the master branch.
> >
> >This last one is a legitimate point that I have considered myself. But I
> >think the value of having tests in HEAD outweigh the cost here.
> >
> >>3. A mix of (1) and (2): we "move" everything under qemu.git, but
> >>keep the compatibility layer and provide a "libqemutest" for
> >>third-parties.
> >>
> >>Anybody writting a test that interacts with qemu will use this
> >>library: qemu, libvirt, spice, ovirt, QE tests, etc.
> >
> >I really see this all as over architecting to be honest.
> >
> >Can someone explain in clear terms (without appealing to maturity,
> >flexibility, or any other qualitative aspect) why it takes anything more
> >than a few dozen shell functions to write all of the tests that are in
> >kvm-autotest test today?
> 
> Clearly as your requirements are different than the ones we had when
> the project was written, qemu-test doesn't need any of the stuff
> present there and you can do it all with a handful of shell script
> functions.
> 
> But to do cover the same things covered in autotest today with the
> same requirements, I really doubt you can do the same with the same
> handful of shell script functions. See the example about the virtio
> test above, not to mention other functionality we have with the
> tests, such as make possible to do tests run with a migration thread
> going on the background, while it's plugging/unplugging devices,
> running a stress test or ltp, or a benchmark, or measuring the time
> drift experienced by the guest.

-- 
Ademar de Souza Reis Jr.
Red Hat

^[:wq!