Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests

[Anthony Liguori]

On 03/08/2012 08:01 AM, Lucas Meneghel Rodrigues wrote:
> On 03/08/2012 10:36 AM, Anthony Liguori wrote:
>
> > > Virt/qemu tests: Minimal guest images
> > > -------------------------------------
> > >
> > > In order to make development level test possible, we need the tests to
> > > run fast.
> > > In order to do that, a set of minimal guest images is being developed
> > > and we
> > > have a version for x86_64 ready and functional:
> > >
> > > https://github.com/autotest/buildroot-autotest
> >
> > I'm really not a fan of buildroot. Note that in order to ship binaries,
> > full source needs to be provided in order to comply with the GPL. The
> > FSF at least states that referring to another website for source that's
> > not under your control doesn't satisfy the requirements of the GPL.
>
> We have a full clone of the buildroot repository that points out to the sources,
> if it's necessary to have a clone of all the projects needed host there in order
> to be able to publish a binary image to help people, then we can do it.

This is harder than I think you anticipate but okay..

> > Just out of curiosity, did you try to use qemu-test? Is there a reason
> > you created something different?
>
> I did, and it does what it proposes to. Nothing against it, but we have code
> that can do more things, that has been developed for longer time.
>
> It's similar to qemu-jeos vs buildbot, you have written scripts to create an
> image, which happens to be precisely why buildroot was written more than 10
> years ago and it works very well, allowing me to put things on the image that
> are not possible with qemu-jeos. If the problem is to point out to all sub
> modules as git repos, we can make that happen too, rather than re-writing stuff
> that works.
>
> For a long time I would like to see people working on a single code base,

I agree, we just disagree on what that code base should be :-)

That code base should be qemu.git.  This discussion isn't about improving 
third-party QE--at least not to me.  Third party QE is a solved problem thanks 
to all of your wonderful work with kvm-autotest.  I'm sure you're looking for 
more participation/developers, but even if you had twice the developers working 
on kvm-autotest, I don't think it would fundamentally change our quality.

I'm interested in driving our development process toward test driven development 
such that all 200+ people that write patches for QEMU for a given release write 
and run tests as part of their normal development process.  The requirements to 
achieve this are different than the requirements you have been working against 
up until now.

Every barrier that we put up to writing and running tests will reduce than 
number of 200+ to something lower.

Submitting a patch to a different project than qemu.git is a barrier.  Now 
instead of getting a single set of feedback, you've got to deal with feedback 
from two projects.

Having to use setup another framework (that runs as root) is another barrier.  I 
change a file in QEMU, run make, then run make check.  I don't install anything, 
I don't sudo anything.  The whole process is relatively quick and painless.

Having to make a change to autotest, then install autotest, relaunch it, etc, is 
just too complicated to be part of a developers fast path.

Now I think we should talk about how to make tests that live in qemu.git and run 
as part of make check easily harnessed by autotest..  But I think the primary 
focus of future test work needs to be within qemu.git.

Regards,

Anthony Liguori

> because that would allow things to progress further and people would have even
> better tools to use.
>
> By implementing the features of qemu-test in autotest we could simply use the
> qemu-test tests and use autotest rather than qemu test, and that's why we have
> done it.