On Thu, Mar 08, 2012 at 11:54:31AM +0000, Stefan Hajnoczi wrote:
On Thu, Mar 8, 2012 at 11:44 AM, Stefan Hajnoczi<address@hidden> wrote:
On Thu, Mar 8, 2012 at 4:00 AM, Lucas Meneghel Rodrigues<address@hidden> wrote:
One of our main goals is to provide useful tools for the qemu community,
since we have a good number of tests and libraries written to perform
integration/QA testing for that tool, being successfuly used by a number of
QA teams that work on qemu. Also, we recently provided a subset of that
infrastructure to test libvirt, one of our virtualization projects of
interest.
Thanks for sharing.
One thing I should have added is that my message is about what it
would take for me to use autotest and contribute tests. But now I
realize that you might be going for a different model:
If you're aiming for a different model where autotest integrates
external test suites (i.e. tests wouldn't be written in autotest.git,
instead autotest.git would contain snapshots of external test suites),
then this proposal seems fine. Upstream projects like QEMU would
develop their own test suite and it would be dropped into autotest or
a specific autotest instance.
Yes, that's the idea. We want autotest to be the framework, not
(just a) collection of tests. We also want each development team
to implement and maintain their own set of tests, using (or not)
the goodies from autotest at their discretion.
In summary, autotest is (or is going to be) a framework that
provides:
- A test runner, with grid/cluster support and advanced
instrumentation
- A devel library and set of utilities for test writers
- A set of pre-built images (JeOS – Just Enough OS) for
test writers
(attached is a picture showing what we want to achieve)
If a project has an internal library or set of utilities that can
be of general use, they can be submitted to autotest.git for
inclusion, thus reaching a broader audience.
A short summary of the plans:
- Tests can live anywhere and each devel team implements and
maintains their own set of tests
- Usage of the autotest library by test writers is optional
- Tests are scripts returning 0 or error (any language)
- Tests can be run individually or in sets
- Tests should run fast, our target is seconds or a few minutes
- The test runner is smart and “just works” by default
- Trivial standard output (FAIL, PASS, SKIPPED)
- Collect logs, OS data and other stuff (e.g. --record-video!)