Hi Philippe, Thomas,
On 11/27/20 6:47 PM, Thomas Huth wrote:
> On 27/11/2020 18.41, Philippe Mathieu-Daudé wrote:
>> We lately realized that the Avocado framework was not designed
>> to be regularly run on CI environments. Therefore, as of 5.2
>> we deprecate the gitlab-ci jobs using Avocado. To not disrupt
>> current users, it is possible to keep the current behavior by
>> setting the QEMU_CI_INTEGRATION_JOBS_PRE_5_2_RELEASE variable
>> (see [*]).
>> From now on, using these jobs (or adding new tests to them)
>> is strongly discouraged.
>>
>> Tests based on Avocado will be ported to new job schemes during
>> the next releases, with better documentation and templates.
>
> Why should we disable the jobs by default as long as there is no replacement
> available yet?
Why keep it enabled if it is failing randomly, if images hardcoded
in tests are being removed from public servers, etc...?
They are not disabled, they are still runnable manually or setting
QEMU_CI_INTEGRATION_JOBS_PRE_5_2_RELEASE...
We realized by default Avocado runs all tests on the CI jobs,
triggering failures and complains. Developer stopped to contribute/
review integration tests because of that. We want developers be
able to contribute tests to the repository fearlessly.
If we don't change anything, we'll keep having CI failures due
to Avocado design issues (artifacts removed from remote servers,
etc...).
I share your concern about the artifacts not being stored on a reliable location that can be used for all Qemu acceptance tests. In particular for the Orange Pi PC tests we have seen it happening, that the image we use was removed from the armbian server.
As a temporary solution perhaps we can use github to store artifacts, until we have a proper location?
I haven't seen any attempt on this list to improve the current
fragile situation, but better suggestions are certainly welcome.
Thanks,
Phil.