Re: [Qemu-devel] The state of testing

[Stefan Hajnoczi]

On Tue, Mar 05, 2013 at 01:21:18PM -0300, Lucas Meneghel Rodrigues wrote:
> On 03/05/2013 01:14 PM, Gerd Hoffmann wrote:
> >   Hi,
> >
> >>>I don't think we have *any* regular autotest coverage for master, have
> >>>we?
> >>
> >>^ Yes, there is coverage. There's a daily job called sanity that takes
> >>about 2.5 hours, that builds the latest kvm.git and installs it in the
> >>host, then reboots the box, builds the latest qemu master and runs:
> >
> >Oh, good.  I guess it would be a good idea to send reports to
> >qemu-devel, maybe a weekly status or mails on regressions simliar to the
> >buildbot build failures.  So people (a) aware that the service exists in
> >the first place and (b) get informed that something broke which should
> >be fixed.
> 
> No doubt, I did try that before, but from my experience, the reports
> are likely to be ignored and people will complain about SPAM.
> 
> So we ended up creating an internal mailing list where Gleb and
> Marcelo are subscribed to, in case they want to look at the reports,
> and I'd look at the reports and report on the ML in case something
> bad happens.

I understand that SPAM issue.  Maybe only send failures to the list?

The reason why it's important to include address@hidden is that
in the current mode you are essentially just reporting bugs.  This is
different from providing maintainers test results and failure
notifications so the tests become part of the QEMU development and
release process.

That's an important distinction.  Maintainers need to see the test
status (e.g. "99/102 tests passing") and have the ability to dig deeper
- especially during the hard freeze period where we shake out bugs.  If
maintainers have no visibility then the tests are not part of the
development process.

The way that buildbot does this is and I think the same is possible for
kvm autotest:
1. Send failure notifications to qemu-devel.
2. Provide a website to check the status
http://buildbot.b1-systems.de/qemu/.

If you want help, please let me know.  qemu.org could host
web-accessible test results.  We can check with Anthony and I can help
you set it up.

Stefan