Re: [Qemu-devel] Please read: make check framework

[Andreas Färber]

Am 09.01.2012 22:42, schrieb Anthony Liguori:
> On 01/09/2012 02:57 PM, Andreas Färber wrote:
>> Am 09.01.2012 20:28, schrieb Anthony Liguori:
>>> We're still short on infrastructure right now so I expect that a lot of
>>> stuff will get merged without tests at first.  As we get more test
>>> infrastructure like qtest and qemu-test, I expect that to change.
>>>
>>> In terms of total coverage for any given feature, I think ultimately the
>>> goal is to move the responsibility for regression testing from the
>>> contributors of new features to the subsystems themselves.
>>>
>>> IOW, after we get going with our test infrastructure, when we do big
>>> sweeping changes like the memory API or QOM, I will have much less
>>> sympathy for breakages that are caused by subsystems with an incomplete
>>> test suite.
>>>
>>> So I think the most important thing for maintainers to start thinking
>>> about is how they can add the necessary infrastructure for testing their
>>> subsystems.
>>
>> Well, I already tried pointing out that qemu-test in the way previously
>> proposed does not help catching the recent regressions for PReP.
> 
> Actually, qemu-test did catch it.  The only reason I didn't see it is
> that I didn't add qemu-system-powerpc to my regression script that I'm
> using to drive it.  But simply booting up a Linux guest would have
> caught it.

Booting up a Linux kernel yes, but not the latest-and-greatest like you
proposed. Meaning, we need to be able to reference different branches or
trees instead of a single submodule (cf. qemu-test thread).

>> The intentions here are certainly good, just the process needs some more
>> thought - or less absoluteness in announcing it.
> 
> There's no magic here.  If you care about your code not breaking, write
> a unit test so that other people can check to see if there changes broke
> your code.
> 
> We can bike shed all day long about how long things should run, or
> whether the wording of the "rules" require us to write tests for
> checkpatch, but it's counter productive.
> 
> If there's some piece of QEMU you care about, start writing tests and
> tie it into make check.  It's that simple :-)

Not quite. It would be easier if we just set up some storage space with
images like the handful of *-test images on qemu.org. That way we can
supply working images for all targets, you can supply build scripts to
rebuild them and we can all be done quickly and have fun.

I was just saying, stop bike-shedding about how we're supposed to deny
patches without test cases from tomorrow on, and instead let's get the
test infrastructure in place and see how well we get along with it.
That's even simpler and much more convenient.

You're building up quite a lot of time pressure on other people lately,
claiming two days for review were long already, wanting to merge QOM and
now make check ASAP - despite all inchoate. Don't expect a lot of
sympathy for ignoring in pursuit of your own agenda that there's other
down- and upstream issues people are handling besides these. :/

Me, I've reviewed most of this series in what I believe is a
constructive way; I'd love to build test cases for my code based on
qtest - you've made this series a prerequisite, so getting libqtest for
writing test cases based on it is going to take some more time until we
can expect contributors to contribute test cases.
If however you're trying to push the work of writing tons of test cases
for legacy code - be it qtest or qemu-test or a qemu-test alternative -
upfront to submaintainers, like you seemed to with this thread, then
you're on the wrong track IMO. KVM on x86_64 has the biggest user base
in terms of people testing and potentially contributing test cases; this
imbalance of manpower is not going to be remedied by any unit test
frameworks we introduce.

Regards,
Andreas