Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests

[Paolo Bonzini]

Il 09/03/2012 14:56, Anthony Liguori ha scritto:
> On 03/09/2012 07:07 AM, Paolo Bonzini wrote:
>> Il 08/03/2012 22:24, Anthony Liguori ha scritto:
>>> The qemu-test tests are smaller than the corresponding autotest 
>>> tests.
>> 
>> They also do much less.
>> 
>> It's true that a combination of qemu-test + qtests could do 99% of 
>> the job more simply than autotest.  But the last 1% (including 
>> migration) would require a large effort, while it would be just 
>> there in autotest.
> 
> Can you clarify here?  When you say "migration", what I think you 
> mean is that kvm-autotest has the ability to run autotest client 
> tests within a guest while migration.

1) Migration is quite different from other tests in terms of VM
lifecycle, even if you use migrate-to-file + restore.  It requires
special infrastructure in the drivers.

2) Migration bugs are subtle and often nondeterministic, so they require
long running tests that do the same thing over and over again.  These
requirements are more similar to integration tests than unit tests.  So
integration tests cannot really be dismissed.

BTW, I don't see the problem with requiring a large "make check" time.
I probably already said this: when I was working on GCC, I ran tests
overnight every time I had piled up enough changes, and it was simply
not possible to skip that step because compiler changes often had
surprising ramifications.  Yes, in some cases I had a bad surprise the
following day, but it's unavoidable.

Even today, bootstrapping and testing GCC on embedded systems can be
done but it takes 4-5 *days*.  Of course you do it first in a
cross-compiler, but surprises still can happen.

But we're lucky: compilers and QEMU are pretty much the opposite, and a
bad surprise should be much more rare here.  QEMU tests and
infrastructure are harder, but the components are delimited much better
than in a compiler with 200 cascading passes, which is what makes qtest
possible at all.

Paolo