Re: QEMU 5.1: Can we require each new device/machine to provided a test?

[Thomas Huth]

On 20/05/2020 01.06, John Snow wrote:
> 
> 
> On 5/19/20 5:04 AM, Daniel P. Berrangé wrote:
>> On Mon, May 18, 2020 at 03:56:36PM -0400, John Snow wrote:
>>>
>>>
>>> On 5/15/20 6:23 AM, Daniel P. Berrangé wrote:
>>>> On Fri, May 15, 2020 at 12:11:17PM +0200, Thomas Huth wrote:
>>>>> On 07/04/2020 12.59, Philippe Mathieu-Daudé wrote:
>>>>>> Hello,
>>>>>>
>>>>>> Following Markus thread on deprecating unmaintained (untested) code
>>>>>> (machines) [1] and the effort done to gather the information shared in
>>>>>> the replies [2], and the various acceptance tests added, is it
>>>>>> feasible to require for the next release that each new device/machine
>>>>>> is provided a test covering it?
>>>>>>
>>>>>> If no, what is missing?
>>>>>
>>>>> If a qtest is feasible, yes, I think we should require one for new
>>>>> devices. But what about machines - you normally need a test image for
>>>>> this. In that case, there is still the question where testing images
>>>>> could be hosted. Not every developer has a web space where they could
>>>>> put their test images onto. And what about images that contain non-free
>>>>> code?
>>>>
>>>> Yep, it isn't feasible to make this a hard rule.
>>>>
>>>> IMHO this is where a support tier classification comes into play
>>>>
>>>>  - Tier 1: actively maintained, qtest coverage available. Expected
>>>>            to work reliably at all times since every commit is CI
>>>>       tested
>>>>
>>>>   - Tier 2: actively maintained, no qtest coverage. Should usually
>>>>            work but regression may creep in due to reliance on the
>>>>       maintainer to manually test on adhoc basis
>>>>
>>>>   - Tier 3: not actively maintained, unknown state but liable to
>>>>             be broken indefinitely
>>>>
>>>> Tier 1 is obviously the most desirable state we would like everthing to
>>>> be at. Contributors will have to fix problems their patches cause as
>>>> they will be blocked by CI.
>>>>
>>>> Tier 2 is an admission that reality gets in the way. Ideally stuff in
>>>> this tier will graduate to Tier 1 at some point. Even if it doesn't
>>>> though, it is still valid to keep it in QEMU long term. Contributors
>>>> shouldn't gratuitously break stuff in these board, but if they do,
>>>> then the maintainer is ultimately responsible for fixing it, as the
>>>> contributors don't have a test rig for it.
>>>>
>>>> Tier 3 is abandonware. If a maintainer doesn't appear, users should
>>>> not expect it to continue to exist long term. Contributors are free
>>>> to send patches which break this, and are under no obligation to
>>>> fix problems in these boards. We may deprecate & delete it after a
>>>> while
>>>>
>>>>
>>>> Over time we'll likely add more criteria to stuff in Tier 1. This
>>>> could lead to some things dropping from Tier 1 to Tier 2. This is
>>>> OK, as it doesn't make those things worse than they already were.
>>>> We're just saying that Tier 2 isn't as thoroughly tested as we
>>>> would like it to be in an ideal world.
>>>
>>> I really like the idea of device support tiers codified directly in the
>>> QEMU codebase, to give upstream users some idea of which devices we
>>> expect to work and which we ... don't, really.
>>>
>>> Not every last device we offer is enterprise production ready, but we
>>> don't necessarily do a good job of explaining which devices fall into
>>> which categories, and we've got quite a few of them.
>>>
>>> I wonder if a 2.5th tier would be useful; something like a "hobbyist"
>>> tier for pet project SoC boards and the like -- they're not abandoned,
>>> but we also don't expect them to work, exactly.
>>>
>>> Mild semantic difference from Tier 3.
>>
>> I guess I was thinking such hobbyist stuff would fall into tier 2  if the
>> hobbyist maintainer actually responds to fixing stuff, or tier 3 if they
>> largely aren't active on the mailing list responding to issues/questions.
>>
>> We add have a 4 tier system overall and put hobbyist stuff at tier 3,
>> and abandonware at tier 4.
>>
>> Probably shouldn't go beyond 4 tiers though, as the more criteria we add
>> the harder it is to clearly decide which tier something should go into.
>>
>> The tier 1 vs 2 divison is clearly split based on CI which is a simple
>> classification to decide on.
>>
>> The tier 2 vs 3 division is moderately clearly split based on whether
>> there is a frequently active maintainer.
>>
>> We can probably squeeze in the 4th tier without too much ambiguity in
>> the classisfication if we think it is adding something worthwhile either
>> from our POV as maintainers, or for users consuming it.
> 
> Yes, I didn't mean to start watering it down into a 1,380 tier system
> that we're never able to properly utilize.
> 
> I was thinking more along the lines of:
> 
> - Device works and is well loved
> - Device works and is well loved (but we have to test manually)
> - Device doesn't work, but is well loved
>   (Use at your own peril, please file a bug report)
> - Device doesn't work, and is unloved
> 
> Perhaps it'd be clearer to name these Tier 1A, 1B, 2, and 3; where
> things can shift from 1A to 1B as their test coverage allows, but it's
> not meant to indicate general status otherwise.

All that sounds somewhat similar to the classification that we already
use in our MAINTAINERS file - Supported, Maintained, Odd-Fixes, Orphan,
Obsolete ... maybe we can avoid to introduce yet another classification
system and merge the two (e.g. by also changing the classification
system in MAINTAINERS a little bit?).

 Thomas