Re: [Qemu-devel] [PULL 10/19] tests/boot_linux_console: increase timeout

[Cleber Rosa]

On Thu, May 09, 2019 at 06:40:40AM +0200, Gerd Hoffmann wrote:
>   Hi,
> 
> > > Tests can also timeout due to slow downloads of test kernels.
> > > Any chance to run the downloads without timeout?
> > 
> > I acknowledge this is an issue, and have thought about two possible
> > ways to solve it:
> > 
> >  1) Downloading/caching/checking all the test assets in a job "pre-tests"
> >     plugin.
> 
> >  2) Report the test phase (say, setUp()) to the test runner, which
> >     would allow the runner to:
> 
> > I'm very much interested in your opinion so we can evolve the idea into
> > implementation.
> 
> (1) is the better approach I think.  That way you can give better
> feedback to the user, along the lines of "download 2 of 5 in progress".
> Also it allows for better management of the assets, you can have tools
> to list them / fetch them / copy them from one to another test machine
> / find & cleanup obsolete assets (i.e. Fedora 28 kernel after switching
> tests to Fedora 29).
> 
> (2) is probably still useful, to report phases of longer running
> tests and maybe have separate timeouts for each phase.  But for
> assets I think approach (1) is better than a "download asset"
> phase.
>

I also think that approach #1 is simpler and saner, but thinking about
where we're going with the test runner development, I started to have
doubts about it.  The reason is that we're adding parallel and multi
environment (process, machine, container) execution capabilities to the
runner. Dogfood version is here:

  https://github.com/avocado-framework/avocado/pull/3111

By doing all download/caching before any single test starts, we may be
wasting a lot of CPU time that could be used for running tests, making
the overall job execution much longer.

And with regards to test phases, the runner could distinguish between
setup and test phases, possibly re-running the setup phase a number of
times until sucessfull completion, improving even further the
reliability of the *test* results (excluding the setup phase).

Anyway, this is only brainstorm at this point, so I'll let you know
when this moves forward.

> cheers,
>   Gerd
> 
> 

FIY: one of the new runner planned features, is based on a question
use case you mention, about how to run tests built into a minimal
image, such as an initrd.

Regards,
- Cleber.