Re: [PATCH v5 1/5] iotests: remove 'linux' from default supported platforms

[Max Reitz]

On 04.10.19 16:05, Peter Maydell wrote:
> On Fri, 4 Oct 2019 at 14:50, Max Reitz <address@hidden> wrote:
>> On 04.10.19 15:16, Peter Maydell wrote:
>>> 'make check' does have the restriction
>>> that we don't want the tests to take too long to run, but in
>>> general the block layer should be running some reasonable subset
>>> of tests in the project's standard "run the tests please" command.
>>> (I have no opinion on exactly what that subset ought to be, beyond
>>> that it would be good if that subset doesn't have known intermittent
>>> failures in it.)
>>
>> Deciding the subset is difficult.  My stance was that it’s better to not
>> choose an arbitrary subset here but ensure that really everything gets
>> run as part of CI, that is asynchronously so it doesn’t block anyone and
>> can take as long as it wants.
> 
> I wonder if we have a terminology difference confusion here.
> To me "CI" means "we have tests which we automatically run
> before pushing commits to master, and if they fail then we
> don't push". So (a) they have to run synchronously and (b) there
> is a need for them to run in a reasonable period of time because
> otherwise it takes too long to run the tests before pushing
> to master and we get a backlog of unprocessed pullrequests
> and annoying delays.

Hm, yes.  In that case, there’s of course no point in moving it.

I meant moving the tests to somewhere where they run asynchronously.

>> If we decide to get pull requests based on that, then that’d bring even
>> more pain, but at least it’d be honest.  But just running half of the
>> qcow2 tests to me seems only like we want to pretend we ran the iotests.
>>  So for me, iotests still break, and we need to deal with make check
>> failures on top.  I’d at least prefer one or the other.
> 
> I think the ideal we're aiming for here is:
>  (1) people doing active work in the block subsystem are probably
> often going to want to run all the iotests, and certainly the
> subsystem maintainers will want to do that as they put together
> pull requests.
>  (2) but people who *don't* do active work in the block subsystem
> still sometimes touch it in passing as part of things like global
> refactorings or other changes that touch big parts of the tree,
> or accidentally break it with a change to some other bit of QEMU
> entirely. These people won't run the full iotests, but it is
> reasonable to expect them to run 'make check', and it would be good
> if that caught "whoops you broke the block subsystem".
> 
> Similarly, "make check" has incredibly spotty coverage of
> various arm boards and devices, but it does do some basic
> checks which do catch accidental breakage.

Yes, it has its use, but at the same time it means that intermittent
failure can appear in make check because some iotest is broken.  (The
past has shown that it’s hard to predict which tests will at some point
start to exhibit such behavior.)

This then requires immediate attention, and I simply want help with that
from someone who really cares about having the test be part of make
check, as I wrote in my reply to Thomas’s patch “iotests: Do not run the
iotests during "make check" anymore”.

The issues are just half as pressing when it isn’t make check that’s
intermittently failing.  (I hope nobody takes this as “He doesn’t want
to fix intermittent failures”, because I do really try to do that.  So I
don’t consider this a good kind of pressure.)


I suppose my main problem is that I’m just lucky that I can’t remember a
case where the block subsystem was really broken and iotests in make
check would have caught it; and that I’m naïve enough to expect this to
continue into the future.

Max

signature.asc
Description: OpenPGP digital signature