Re: [Qemu-devel] Status and RFC of patchew testings on QEMU

[Daniel P. Berrange]

On Mon, Jul 17, 2017 at 11:41:38AM +0200, Thomas Huth wrote:
> On 17.07.2017 08:35, Fam Zheng wrote:
> > Hi all,
> > 
> > Today I've included a fourth type of the automatic patchew replies: FreeBSD.
> > 
> > So far we have these tests running by patchew on each patch series:
> > 
> >   * Docker tests
> >     Basically it is
> >         make address@hidden \
> >              address@hidden \
> >              address@hidden"
> > 
> >   * checkpatch.pl
> >     Each patch is fed to ./scripts/checkpatch.pl and all errors are 
> > reported.
> > 
> >   * s390x
> >     It runs on a machine shared by Fedora team, basically only "./configure 
> > and
> >     make", because "make check" hanging is tricky to deal with from an
> >     automation perspective. (Ideas?)
> 
> Is there any check that could hang "forever"? I think most of the checks
> should have a proper timeout of one or two minutes, don't they?
> 
> Maybe you could also simply run the "make check" with the "timeout"
> command to avoid that it hangs forever?
> 
> >   * FreeBSD
> >     Like s390x.
> > 
> > Q1: In the worst case, you get four individual auto replies from patchew. Is
> > that too many? Do you prefer one reply with all the results concatenated 
> > into
> > one?
> 
> One result would be nicer, I think.
> 
> > Q2: Some think the full log in the mail body is more than necessary. Is it
> > better or worse if it is a "tail -n 200" of the log in the body and the 
> > full log
> > attached?
> > 
> > Q3: What other tests do maintainers want? Different hosts? Different 
> > configure
> > combinations?
> 
> Not sure, but should we maybe also check compiling with the configure
> switches set to non-default values? E.g. --enable-debug --disable-slirp
> --enable-tcg-interpreter --disable-tcg etc. ?

I'm a little concerned about the fact that we've now got three different
sets of tests that are being run on pull requests. There are the tests
that Peter runs on various combinations at time of merge, the tests run 
by patchw at time of submissions, and the tests run by travis after
merge. Each of them is covering a different set of scenarios with only
partial overlap between them.

As a maintainer sending pull requests, I want to try to run all relevant
tests before sending a PR, to minimize chance of it being rejected. It
is hard to come up with a workflow that maximises my coverage across all
three test systems that are run.

I mostly trigger running of the travis tests by pushing to a github banch 
in my qemu clone, but that misses coverage of things done by patchw and by
Peter, sometimes requiring many re-submissions of a PR before all three 
test processes pass. I find this is quite frustrating & a time sink for
everyone involved.

My ideal view of the world would be a single testing system which we
feed with jobs from different places. eg patchw can feed it mail submissions,
Peter can feed it candidate merges before pushing to master, something can
feed it git master after push, and/or periodically, and contributors can
feed it personal branches. IOW, no matter who/what triggers the tests, we
always run the exact same set of tests.  BUilding and maintaining such a
system is hard work, potentially expensive (in terms of hardware required),
and an ongoing full time job for at least one person, ideally more. So I
realize I'm asking for magical dancing ponies here, but it is nice to be 
able to dream.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|