Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests

[Anthony Liguori]

On 03/08/2012 04:24 PM, Ademar Reis wrote:
> On Thu, Mar 08, 2012 at 03:24:15PM -0600, Anthony Liguori wrote:
> > On 03/08/2012 03:02 PM, Ademar Reis wrote:
> > > On Thu, Mar 08, 2012 at 01:16:58PM -0600, Anthony Liguori wrote:
> > > > On 03/08/2012 11:59 AM, Ademar Reis wrote:
> > >
> > > <snip>
> > >
> > > > > > I expect QEMU to grow tests for anything that involves launching
> > > > > > QEMU directly.  Where I would not see QEMU growing tests for is
> > > > > > things like launching QEMU through libvirt.
> > > > >
> > > > > Agree. For QEMU developers, libvirt should not be on the way, the
> > > > > interaction should be minimal or non-existent.
> > > > >
> > > > > That's an area which will require some work in libautotest,
> > > > > because due to previous QE requirements, it now invokes libvirt
> > > > > methods instead of QEMU directly for a lot of stuff. But it can
> > > > > be done and is part of the plan.
> > > >
> > > > If you're talking about libautotest as an abstraction layer for
> > > > launching QEMU, I don't think that's something we should use in
> > > > upstream QEMU.  Abstraction layers are okay when the things you are
> > > > abstracting are well defined and relatively static.  We're talking
> > > > about testing the tip of a development tree though and coordinating
> > > > changes with an abstraction layer would be counter productive.
> > >
> > > Valid point. If I'm a qemu developer and I want to make huge
> > > changes, ideally I shouldn't have to submit patches to
> > > libautotest because the tests in qemu.org will break.
> > >
> > > I don't know how likely such scenario is (Lucas is maitaining
> > > kvm-autotest for years now and the integration hardly breaks)
> > > But we do have a conflict of interests, so lets evaluate a
> > > few options:
> >
> > Because kvm-autotest doesn't have thousands of tests and isn't part
> > of 200+ developers fast paths :-)
> >
> > > My main interests are:
> > >      - remove barriers for developers to write tests
> >
> > This is my number one priority.
> >
> > >      - share the testing effort between multiple projects (qemu,
> > >        libvirt, ovirt, spice, QE, etc...)
> >
> > This is my number 99 priority.
>
> You have the right to consider the interests of qemu only, but
> qemu is used by several other teams who depend on it to implement
> their own tests. Some collaboration would be extremely beneficial
> for everybody.
>
> Collaboration goes both ways: you get benefits from the
> interaction and code reuse with different projects and teams, but
> it may require a few tradeoffs.

I think the problem here effectively boils down to "code sharing at any cost".

Let's look at it in purely practical terms.  How many lines of code would be 
saved here and what does that code do?

If the only sharing is some infrastructure bits in libautotest, then I suspect 
we're talking in the low 10ks of lines at the absolute maximum (and I don't 
think it's even close to that in reality).  Launching QEMU and connecting to QMP 
isn't that hard.  It's in the hundreds of lines of code.  What else is there 
that's need to write a test?

I don't think this is sufficiently compelling to add an unconditional autotest 
dependency to QEMU.

> > I think you could achieve practical code sharing by decomposing
> > kvm-autotest into a set of useful, independent tools.  For
> > instance....
> >
> > > Now the options:
> > >
> > > 1. We keep the invocation stuff in libautotest and when writting
> > > a complex test in qemu.git, the test writter can choose to use it
> > > because of the goodies there.
> > >
> > > Pros:
> > >    - Lots of codepaths implemented both in python and as cmd-line
> > >      utilities: less lines of code to write a test, smaller
> > >      barrier for the developer.
> >
> > I've got an existence proof that this is not true.  The qemu-test
> > tests are smaller than the corresponding autotest tests.
>
> You're comparing developer-level tests with the existent QA-level
> tests (much more complex).

Let's be specific then.  Look at device-add.sh in qemu-test.  It's 71LOC. 
pci_hotplug.py in autotest is 204LOC.

What's the "much more complex" part of pci_hotplug.py that is enabled by having 
the autotest framework?

> We're talking about an approach where you'll need just a few
> lines of code to instantiate a VM and interact with it.

This isn't that hard of a problem.  It just takes a few lines to launch QEMU in 
qemu-test too.

> > >    - Mature code in the test library.
> > >    - Goodies such as video-recording and system info collection
> > >      in a standard way for different projects.
> >
> > Why does video-recording have to be part of libautotest?  Why can't
> > you just have a simply utility that connects to a VNC session and
> > records it?  Presumably that's all it's doing here.
> >
> > Likewise, there are dozens of "system info collection" such as
> > sosreport..  why fold this all into libautotest?
>
> For standardization: think outside of the qemu community. A lot
> of our developers contribute code to multiple projects. It gets
> easier if there's a standard way to achieve the same results when
> writting tests for different projects.
>
> Anyway, it's a minor issue.

Yeah, I think this is outside of the scope of the current discussion as it's 
sort of like trying to boil the ocean :-)

> > >    - Code shared with other teams.
> > >    - The same test code can run on old platforms because
> > >      libautotest has a compatibility layer, thus enabling a
> > >      larger grid of tests.
> >
> > This is not a requirement from a QEMU perspective nor do I think it's useful.
>
> Well, it's not a requirement, but you can't expect QE teams to
> spend their time on tests that work only on HEAD. It makes no
> sense from their point of view.

They're certainly testing some sort of HEAD, even if it's not qemu.git.  I don't 
see any conflicting goals here.  Write a test against upstream, backport to a 
downstream if necessarily.

> > >    - QE teams will be closer to the development, sharing
> > >      some of their testing code and maintenance burden.
> >
> > The QE team could simply contribute tests to qemu.git to achieve this same goal.
>
> They won't help you if you don't help them. Again, it doesn't
> make sense for QE to work on HEAD. At least not the QE teams I'm
> talking about.
>
> And I say from my own experience: maintaining a set of thousands
> of tests is far from trivial. You'll waste a lot of potential
> contributions by alienating the fully-dedicated teams currently
> working on QE.

And trying to maintain a thousand tests against any possible version is probably 
also unrealistic.

Is the expectation that all of the tests in WebKit work against any possible 
version of WebKit?  How do they enforce this since it's all in trunk?

> > > 2. We "move" (or implement) all the libautotest stuff related to
> > > qemu invocation and interaction into qemu.git, forbidding the
> > > dependency of autotest in qemu.git.
> >
> > Let me just be really clear here.  In order to do TDD, tests have to
> > be a mandatory part of the QEMU build process.  We cannot introduce
> > new dependencies in order to build tests.
> >
> > Making autotest an unconditional dependency of qemu.git is a non-starter for me.
>
> You have dependencies on gtest and other libraries. it's not that
> different.

We have a dependency on glib, which gtest happens to be a part of.  It was a big 
deal introducing that dependency and gtest was not the only (or primary) reason 
to introduce that dependency.

> Plus it's not unconditional: the test runner will report tests
> SKIPPED if a dependency is not present.

But then the tests aren't run so if most developers didn't have it installed, 
and most tests were written with it, most developers wouldn't be running most 
tests which defeats the purpose, no?

> It's actually the oposite: you'll have to forbid developers from
> submiting tests that use libautotest if you don't want the
> dependency.

I think part of the problem here is the abstract nature of this discussion. 
What is libautotest actually supposed to do?

This would be an easier discussion if we were talking about patches instead of 
proposals.

> > >    - QE will be alienated from the qemu test effort. There will be
> > >      no integration between the QE efforts and the maintenance of
> > >      the qemu developer-level tests.
> >
> > I think we're a pretty friendly and open community :-)  There is no
> > reason that QE should be "alienated" unless folks are choosing not
> > to participate upstream.
>
> For the exact same reasons you as a developer don't want to
> implement tests inside autotest, QE won't want to implement tests
> for qemu.git. It's out of their comfort zone, just put yourself
> on their shoes.

This is a really, really poor argument and I hope I don't need to go into 
details of why.  If the primary reason for libautotest is so the people writing 
tests for QEMU can avoid actually working with the developers of QEMU...  we've 
got a problem.

> > >    - You don't get the goodies from autotest (standardized system
> > >      info collection, video recording, grid support, etc).
> > >    - The new tests will work only on the master branch.
> >
> > This last one is a legitimate point that I have considered myself.
> > But I think the value of having tests in HEAD outweigh the cost
> > here.
>
> Thinking about qemu only, agree.
>
> > > 3. A mix of (1) and (2): we "move" everything under qemu.git, but
> > > keep the compatibility layer and provide a "libqemutest" for
> > > third-parties.
> > >
> > > Anybody writting a test that interacts with qemu will use this
> > > library: qemu, libvirt, spice, ovirt, QE tests, etc.
> >
> > I really see this all as over architecting to be honest.
> >
> > Can someone explain in clear terms (without appealing to maturity,
> > flexibility, or any other qualitative aspect) why it takes anything
> > more than a few dozen shell functions to write all of the tests that
> > are in kvm-autotest test today?
>
> This is close to being offensive IMO, so I won't answer. :-)

I honestly don't mean it to be so I apologize if it comes off that way.  But I 
dislike arguing in such abstract terms.  What are the features that you think 
libautotest is needed for?

> > You launch qemu (by just using a command), and you interact with the
> > monitor via QMP and HMP.  Why do you ever need anything more than
> > that?
>
> Well, again, I'll agree to disagree.
>
> > Why does libautotest even need to exist?
>
> You're looking at things from the sole point of view of a qemu
> developer. If you don't want to see things from a different
> perspective and accept any kind of tradeoffs for better
> collaboration with other teams, then unfortunately there's not
> much we can do. It's a pity, because collaboration goes both
> ways.

I need something more concrete then this.  What is it about libautotest that 
would allow collaboration?

> We're trying very hard to satisfy your requirements and
> interests. Don't you see any way you can help us satisfy
> ours?

I'm having a hard time understanding what your requirements are other than that 
we should write our tests using libautotest (which has not really been explained 
other than being levied as a requirement).

Regards,

Anthony Liguori