Re: [Qemu-devel] QEMU Summit 2017: minutes

[Thomas Huth]

On 27.11.2017 23:03, John Snow wrote:
> 
> On 11/23/2017 11:31 AM, Peter Maydell wrote:
[...]
>> Continuous Integration:
>>  * Christian Borntraeger: qemu-iotests have broken a lot, they should be
>>    run before patches are merged
> 
> This, rather unfortunately, is a huge testing burden. I try to make sure
> I do it for everything I submit, but for the volume of block patches it
> really does rely CI. The more we add (to our pitifully sparse iotesting,
> I might add) the longer it takes. Ensuring per-patch testing begins to
> take prohibitively long.
> 
> Perhaps per-pull or per-merge becomes more feasible. Maybe if we do
> implement a block-next amalgam we'd be able to batch our testing on a
> weekly basis.

I think you block-layer folks should do at least run the qemu-iotests
before sending a pull request to Peter. The iotests should really not be
broken in upstream master.

>>  * Peter Maydell: If it isn't tested by "make check" then it isn't tested:
>>    so if something is regularly regressing then it needs to be added to
>>    "make check".
> 
> Is this tenable long term? We can't conceivably state that we will never
> test things that aren't in "make check" -- we ought to have different
> tiers, at least. The full testing suite should run for RC tags at least,
> but it's not feasible (I think?) to run the entire battery of tests on
> every commit... but that shouldn't stop us from running them /sometimes/...

We've already got "make check SPEED=slow" for running tests that take a
lot of time. So maybe you could do that in the iotests as well, so that
the normal, quick tests can be run during "make check" and the full
iotest suite is only run during "make check SPEED=slow" ?

 Thomas