[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [Qemu-devel] [PATCH v3 6/7] tests/qemu-iotests/group: Re-use the "au
From: |
Markus Armbruster |
Subject: |
Re: [Qemu-devel] [PATCH v3 6/7] tests/qemu-iotests/group: Re-use the "auto" group for tests that can always run |
Date: |
Fri, 10 May 2019 17:29:42 +0200 |
User-agent: |
Gnus/5.13 (Gnus v5.13) Emacs/26.1 (gnu/linux) |
Kevin Wolf <address@hidden> writes:
> Am 10.05.2019 um 10:55 hat Thomas Huth geschrieben:
>> On 08/05/2019 07.47, Thomas Huth wrote:
>> > On 07/05/2019 17.50, Eric Blake wrote:
>> >> On 5/7/19 10:22 AM, Thomas Huth wrote:
>> >>> On 07/05/2019 15.22, Markus Armbruster wrote:
>> >>>> Thomas Huth <address@hidden> writes:
>> >>>>
>> >>>>> Currently, all tests are in the "auto" group. This is a little bit
>> >>>>> pointless.
>> >>>>> OTOH, we need a group for the tests that we can automatically run
>> >>>>> during
>> >>>>> "make check" each time, too. Tests in this new group are supposed to
>> >>>>> run
>> >>>>> with every possible QEMU configuration, for example they must run with
>> >>>>> every
>> >>>>> QEMU binary (also non-x86), without failing when an optional features
>> >>>>> is
>> >>>>> missing (but reporting "skip" is ok), and be able to run on all kind
>> >>>>> of host
>> >>>>> filesystems and users (i.e. also as "nobody" or "root").
>> >>>>> So let's use the "auto" group for this class of tests now. The initial
>> >>>>> list has been determined by running the iotests with non-x86 QEMU
>> >>>>> targets
>> >>>>> and with our CI pipelines on Gitlab, Cirrus-CI and Travis (i.e.
>> >>>>> including
>> >>>>> macOS and FreeBSD).
>> >>>>
>> >>>> I wonder whether we should additionally limit "make check" to "quick"
>> >>>> tests. How slow are the non-quick auto tests for you?
>> >>>
>> >>> I already sorted out some of the tests that run veeeery long, since the
>> >>> run time on gitlab, cirrus-ci and travis is limited. "make check-block"
>> >>> currently takes 3 minutes on my laptop, I think that's still ok?
>> >>>
>> >>> When I run the tests from the auto group that are not in the quick
>> >>> group, I currently get:
>> >>>
>> >>
>> >> My personal threshold is about 5 seconds for quick, so:
>> >>
>> >>> 003 1s ...
>> >>> 007 2s ...
>> >>
>> >> Should these be moved to quick?
>> >
>> > I'll leave that decision up to the blocklayer folks ... I thought that
>> > there might have been a different reason that these have not been put
>> > into "quick" yet...?
>> >
>> >>> 013 5s ...
>> >>
>> >> this one is borderline
>> >>
>> >>> 014 15s ...
>> >>> 015 9s ...
>> >>
>> >> Definitely not quick, but if you think they are still okay for auto, I
>> >> can live with that.
>> >>
>> >>> 022 1s ...
>> >>
>> >> Another candidate for quick?
>> >>
>> >>> 023 18s ...
>> >>
>> >> Even longer than 14. Okay for auto?
>> >
>> > I think I'd give it a try. If people are complaining later that "make
>> > check" is running now way too long, we still can refine the list later.
>>
>> Thinking about this again, "make check" now runs quite a bit longer
>> indeed. So I now rather tend to remove the tests that run longer than 5s
>> from the auto group instead... I think I'll send a v4 of this patch
>> where I'll remove them from the auto group.
>
> I don't think time is everything. We should also consider how much
> the tests contribute to basic code coverage. There is no point in
> removing a test from the list because it takes 10 seconds, but if I
> split it in two tests taking each 5 seconds, you would include both
> halves.
>
> For example, 030, 040 and 041 are not that quick (14/11/42 seconds,
> respectively), but they are the most important tests for block jobs and
> covering a lot. Sure, 42 seconds is a lot, but I'd keep 030 and 040 at
> least.
Yes, we want block jobs covered. However, 42 seconds is a lot indeed.
Can you think of ways to get a useful part of the full coverage in five
seconds or less?
- [Qemu-devel] [PATCH v3 3/7] tests/qemu-iotests: Do not hard-code the path to bash, (continued)
- [Qemu-devel] [PATCH v3 3/7] tests/qemu-iotests: Do not hard-code the path to bash, Thomas Huth, 2019/05/02
- [Qemu-devel] [PATCH v3 6/7] tests/qemu-iotests/group: Re-use the "auto" group for tests that can always run, Thomas Huth, 2019/05/02
- Re: [Qemu-devel] [PATCH v3 6/7] tests/qemu-iotests/group: Re-use the "auto" group for tests that can always run, Markus Armbruster, 2019/05/07
- Re: [Qemu-devel] [PATCH v3 6/7] tests/qemu-iotests/group: Re-use the "auto" group for tests that can always run, Thomas Huth, 2019/05/07
- Re: [Qemu-devel] [PATCH v3 6/7] tests/qemu-iotests/group: Re-use the "auto" group for tests that can always run, Eric Blake, 2019/05/07
- Re: [Qemu-devel] [PATCH v3 6/7] tests/qemu-iotests/group: Re-use the "auto" group for tests that can always run, Thomas Huth, 2019/05/08
- Re: [Qemu-devel] [PATCH v3 6/7] tests/qemu-iotests/group: Re-use the "auto" group for tests that can always run, Thomas Huth, 2019/05/10
- [Qemu-devel] [PATCH v4] tests/qemu-iotests/group: Re-use the "auto" group for tests that can always run, Thomas Huth, 2019/05/10
- Re: [Qemu-devel] [PATCH v3 6/7] tests/qemu-iotests/group: Re-use the "auto" group for tests that can always run, Markus Armbruster, 2019/05/10
- Re: [Qemu-devel] [PATCH v3 6/7] tests/qemu-iotests/group: Re-use the "auto" group for tests that can always run, Kevin Wolf, 2019/05/10
- Re: [Qemu-devel] [PATCH v3 6/7] tests/qemu-iotests/group: Re-use the "auto" group for tests that can always run,
Markus Armbruster <=
- Re: [Qemu-devel] [PATCH v3 6/7] tests/qemu-iotests/group: Re-use the "auto" group for tests that can always run, Kevin Wolf, 2019/05/10
- Re: [Qemu-devel] [PATCH v3 6/7] tests/qemu-iotests/group: Re-use the "auto" group for tests that can always run, Markus Armbruster, 2019/05/08
[Qemu-devel] [PATCH v3 4/7] cirrus / travis: Add gnu-sed and bash for macOS and FreeBSD, Thomas Huth, 2019/05/02
[Qemu-devel] [PATCH v3 7/7] tests: Run the iotests during "make check" again, Thomas Huth, 2019/05/02