Re: [Qemu-devel] Status and RFC of patchew testings on QEMU

[Daniel P. Berrange]

On Mon, Jul 17, 2017 at 12:00:22PM +0100, Peter Maydell wrote:
> On 17 July 2017 at 11:20, Daniel P. Berrange <address@hidden> wrote:
> > I'm a little concerned about the fact that we've now got three different
> > sets of tests that are being run on pull requests. There are the tests
> > that Peter runs on various combinations at time of merge, the tests run
> > by patchw at time of submissions, and the tests run by travis after
> > merge. Each of them is covering a different set of scenarios with only
> > partial overlap between them.
> 
> I agree -- I have my ad-hoc test system not because I enjoy
> collecting random things to run tests on but because we don't
> have an automated system that does what I need:
>  * wide coverage, focusing on oddball host architectures and OSes
>  * ability to ssh in to a machine where tests have broken if need be
>  * decently fast to complete a test run (travis is terrible for this)
> 
> Note that "fast to complete a test run for me" is in conflict with
> "usable by anybody who wants to submit a pull request" :-)

Not neccessarily. It does mean though that if we had an automated test
system, we would have to reserve some portion of hardware resources for
exclusively jobs triggered by the person responsible for git master and
stable branch merges, so that it can never get delayed by a backlog of
jobs from the mailing list.

OpenStack has this kind of CI system - they have 2 separate job queues,
one testing all patches submitted for code review, and one queue only
processing patches for merge, so the former can't negatively impact the
progress of the latter. Availability of (money/sponsorship for) hardware
though is an obvious limiting factor here.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|