Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests

[Stefan Hajnoczi]

On Thu, Mar 8, 2012 at 11:51 PM, Ademar Reis <address@hidden> wrote:
> On Thu, Mar 08, 2012 at 05:21:44PM -0600, Anthony Liguori wrote:
>> On 03/08/2012 04:24 PM, Ademar Reis wrote:
>> >On Thu, Mar 08, 2012 at 03:24:15PM -0600, Anthony Liguori wrote:
>> >>On 03/08/2012 03:02 PM, Ademar Reis wrote:
>> >>>On Thu, Mar 08, 2012 at 01:16:58PM -0600, Anthony Liguori wrote:
>> >>>>On 03/08/2012 11:59 AM, Ademar Reis wrote:
>> >>>   - QE will be alienated from the qemu test effort. There will be
>> >>>     no integration between the QE efforts and the maintenance of
>> >>>     the qemu developer-level tests.
>> >>
>> >>I think we're a pretty friendly and open community :-)  There is no
>> >>reason that QE should be "alienated" unless folks are choosing not
>> >>to participate upstream.
>> >
>> >For the exact same reasons you as a developer don't want to
>> >implement tests inside autotest, QE won't want to implement tests
>> >for qemu.git. It's out of their comfort zone, just put yourself
>> >on their shoes.
>>
>> This is a really, really poor argument and I hope I don't need to go
>> into details of why.  If the primary reason for libautotest is so
>> the people writing tests for QEMU can avoid actually working with
>> the developers of QEMU...  we've got a problem.
>
> No, one of the benefits of having libautotest is to *collaborate*
> with QE. I'll explain again:
>
> - As a qemu developer, I don't want to spend my time learning and
>  getting involved in autotest, which is a complex QE project
>  (I heard this numerous times).
>
> - As a Quality Engineer, I don't want to invest my time learning
>  and getting involved into upstream qemu to test HEAD.

I think this is the key point of the whole discussion - most of the
other topics have been distractions.  Both communities do testing but
we test different things and have different priorities.

For me this has been the big realization from this discussion.  I felt
kvm-autotest and qemu should share tests.  I was pushing for that but
after following this thread I don't think it makes sense, here's why:

The Quality Engineer you describe is not a QEMU upstream QE, instead
the QE has a broader and more downstream focus.  (This is why
comparisons with WebKit or other upstream projects doing testing are
not valid comparisons.)

There is not enough in common between upstream QEMU testing and
downstream KVM QE to make convergence a win-win.  libautotest sounds
like a technical solution to a people problem - the problem is that we
have different priorities.  We overlap but at the end of the day we do
different things.  We can make a best effort to converge but I don't
see incentives that will make this a success.  Creating an abstraction
library will be sub-optimal for both communities.

I think what's much more valuable is for qemu.git tests to be easily
hooked into autotest.  That way you get access to the testing that the
qemu community is doing for free.

And on the flip-side it would be awesome for kvm-autotest to cover
upstream.  As an example, QEMU uses buildbot with a public web
interface which shows the history of all runs and failure
notifications are sent to qemu-devel (it's not perfect but I think it
adds value to the community).  Can you can do daily/weekly qemu.git
upstream kvm-autotest runs, make the results easily accessible on the
public web, and perhaps hook into the qemu-devel mailing list?

That level of collaboration would allow both communities to do what
they want to do effectively, while still getting benefits from each
other.

Stefan