Re: [Qemu-block] [Qemu-devel] [PATCH v3 6/7] tests/qemu-iotests/group: Re-use the "auto" group for tests that can always run

[Thomas Huth]

On 07/05/2019 15.22, Markus Armbruster wrote:
> Thomas Huth <address@hidden> writes:
> 
>> Currently, all tests are in the "auto" group. This is a little bit pointless.
>> OTOH, we need a group for the tests that we can automatically run during
>> "make check" each time, too. Tests in this new group are supposed to run
>> with every possible QEMU configuration, for example they must run with every
>> QEMU binary (also non-x86), without failing when an optional features is
>> missing (but reporting "skip" is ok), and be able to run on all kind of host
>> filesystems and users (i.e. also as "nobody" or "root").
>> So let's use the "auto" group for this class of tests now. The initial
>> list has been determined by running the iotests with non-x86 QEMU targets
>> and with our CI pipelines on Gitlab, Cirrus-CI and Travis (i.e. including
>> macOS and FreeBSD).
> 
> I wonder whether we should additionally limit "make check" to "quick"
> tests.  How slow are the non-quick auto tests for you?

I already sorted out some of the tests that run veeeery long, since the
run time on gitlab, cirrus-ci and travis is limited. "make check-block"
currently takes 3 minutes on my laptop, I think that's still ok?

When I run the tests from the auto group that are not in the quick
group, I currently get:

003 1s ...
007 2s ...
013 5s ...
014 15s ...
015 9s ...
022 1s ...
023 18s ...
043 0s ...
049 3s ...
061 4s ...
079 2s ...
080 4s ...
091 1s ...
097 2s ...
104 0s ...
117 0s ...
122 8s ...
126 1s ...
137 1s ...
142 5s ...
172 3s ...
174 0s ...
176 5s ...
181 2s ...
186 2s ...
187 0s ...
191 5s ...
201 2s ...
214 0s ...

That looks reasonable to me.

 Thomas