Re: QEMU 5.1: Can we require each new device/machine to provided a test?

[Daniel P . Berrangé]

On Mon, May 18, 2020 at 03:56:36PM -0400, John Snow wrote:
> 
> 
> On 5/15/20 6:23 AM, Daniel P. Berrangé wrote:
> > On Fri, May 15, 2020 at 12:11:17PM +0200, Thomas Huth wrote:
> >> On 07/04/2020 12.59, Philippe Mathieu-Daudé wrote:
> >>> Hello,
> >>>
> >>> Following Markus thread on deprecating unmaintained (untested) code
> >>> (machines) [1] and the effort done to gather the information shared in
> >>> the replies [2], and the various acceptance tests added, is it
> >>> feasible to require for the next release that each new device/machine
> >>> is provided a test covering it?
> >>>
> >>> If no, what is missing?
> >>
> >> If a qtest is feasible, yes, I think we should require one for new
> >> devices. But what about machines - you normally need a test image for
> >> this. In that case, there is still the question where testing images
> >> could be hosted. Not every developer has a web space where they could
> >> put their test images onto. And what about images that contain non-free
> >> code?
> > 
> > Yep, it isn't feasible to make this a hard rule.
> > 
> > IMHO this is where a support tier classification comes into play
> > 
> >  - Tier 1: actively maintained, qtest coverage available. Expected
> >            to work reliably at all times since every commit is CI
> >        tested
> > 
> >   - Tier 2: actively maintained, no qtest coverage. Should usually
> >            work but regression may creep in due to reliance on the
> >        maintainer to manually test on adhoc basis
> > 
> >   - Tier 3: not actively maintained, unknown state but liable to
> >             be broken indefinitely
> > 
> > Tier 1 is obviously the most desirable state we would like everthing to
> > be at. Contributors will have to fix problems their patches cause as
> > they will be blocked by CI.
> > 
> > Tier 2 is an admission that reality gets in the way. Ideally stuff in
> > this tier will graduate to Tier 1 at some point. Even if it doesn't
> > though, it is still valid to keep it in QEMU long term. Contributors
> > shouldn't gratuitously break stuff in these board, but if they do,
> > then the maintainer is ultimately responsible for fixing it, as the
> > contributors don't have a test rig for it.
> > 
> > Tier 3 is abandonware. If a maintainer doesn't appear, users should
> > not expect it to continue to exist long term. Contributors are free
> > to send patches which break this, and are under no obligation to
> > fix problems in these boards. We may deprecate & delete it after a
> > while
> > 
> > 
> > Over time we'll likely add more criteria to stuff in Tier 1. This
> > could lead to some things dropping from Tier 1 to Tier 2. This is
> > OK, as it doesn't make those things worse than they already were.
> > We're just saying that Tier 2 isn't as thoroughly tested as we
> > would like it to be in an ideal world.
> 
> I really like the idea of device support tiers codified directly in the
> QEMU codebase, to give upstream users some idea of which devices we
> expect to work and which we ... don't, really.
> 
> Not every last device we offer is enterprise production ready, but we
> don't necessarily do a good job of explaining which devices fall into
> which categories, and we've got quite a few of them.
> 
> I wonder if a 2.5th tier would be useful; something like a "hobbyist"
> tier for pet project SoC boards and the like -- they're not abandoned,
> but we also don't expect them to work, exactly.
> 
> Mild semantic difference from Tier 3.

I guess I was thinking such hobbyist stuff would fall into tier 2  if the
hobbyist maintainer actually responds to fixing stuff, or tier 3 if they
largely aren't active on the mailing list responding to issues/questions.

We add have a 4 tier system overall and put hobbyist stuff at tier 3,
and abandonware at tier 4.

Probably shouldn't go beyond 4 tiers though, as the more criteria we add
the harder it is to clearly decide which tier something should go into.

The tier 1 vs 2 divison is clearly split based on CI which is a simple
classification to decide on.

The tier 2 vs 3 division is moderately clearly split based on whether
there is a frequently active maintainer.

We can probably squeeze in the 4th tier without too much ambiguity in
the classisfication if we think it is adding something worthwhile either
from our POV as maintainers, or for users consuming it.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|