[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Qemu-devel] [PATCH v3 6/7] tests/qemu-iotests/group: Re-use the "au

From: Markus Armbruster
Subject: Re: [Qemu-devel] [PATCH v3 6/7] tests/qemu-iotests/group: Re-use the "auto" group for tests that can always run
Date: Wed, 08 May 2019 14:46:15 +0200
User-agent: Gnus/5.13 (Gnus v5.13) Emacs/26.1 (gnu/linux)

Eric Blake <address@hidden> writes:

> On 5/7/19 10:22 AM, Thomas Huth wrote:
>> On 07/05/2019 15.22, Markus Armbruster wrote:
>>> Thomas Huth <address@hidden> writes:
>>>> Currently, all tests are in the "auto" group. This is a little bit 
>>>> pointless.
>>>> OTOH, we need a group for the tests that we can automatically run during
>>>> "make check" each time, too. Tests in this new group are supposed to run
>>>> with every possible QEMU configuration, for example they must run with 
>>>> every
>>>> QEMU binary (also non-x86), without failing when an optional features is
>>>> missing (but reporting "skip" is ok), and be able to run on all kind of 
>>>> host
>>>> filesystems and users (i.e. also as "nobody" or "root").
>>>> So let's use the "auto" group for this class of tests now. The initial
>>>> list has been determined by running the iotests with non-x86 QEMU targets
>>>> and with our CI pipelines on Gitlab, Cirrus-CI and Travis (i.e. including
>>>> macOS and FreeBSD).
>>> I wonder whether we should additionally limit "make check" to "quick"
>>> tests.  How slow are the non-quick auto tests for you?
>> I already sorted out some of the tests that run veeeery long, since the
>> run time on gitlab, cirrus-ci and travis is limited. "make check-block"
>> currently takes 3 minutes on my laptop, I think that's still ok?
>> When I run the tests from the auto group that are not in the quick
>> group, I currently get:
> My personal threshold is about 5 seconds for quick, so:
>> 003 1s ...
>> 007 2s ...
> Should these be moved to quick?
>> 013 5s ...
> this one is borderline
>> 014 15s ...
>> 015 9s ...
> Definitely not quick, but if you think they are still okay for auto, I
> can live with that.
>> 022 1s ...
> Another candidate for quick?
>> 023 18s ...
> Even longer than 14. Okay for auto?
> etc.

Kevin, Max?

reply via email to

[Prev in Thread] Current Thread [Next in Thread]