Re: [PATCH v4 1/6] tests/requirements.txt: bump up avocado-framework version to 101.0

[Cleber Rosa]

On 4/17/23 12:50, John Snow wrote:
> On Mon, Apr 17, 2023 at 9:43 AM Alex Bennée <alex.bennee@linaro.org> wrote:
> > From: Kautuk Consul <kconsul@linux.vnet.ibm.com>
> >
> > Avocado version 101.0 has a fix to re-compute the checksum
> > of an asset file if the algorithm used in the *-CHECKSUM
> > file isn't the same as the one being passed to it by the
> > avocado user (i.e. the avocado_qemu python module).
> > In the earlier avocado versions this fix wasn't there due
> > to which if the checksum wouldn't match the earlier
> > checksum (calculated by a different algorithm), the avocado
> > code would start downloading a fresh image from the internet
> > URL thus making the test-cases take longer to execute.
> >
> > Bump up the avocado-framework version to 101.0.
> >
> > Signed-off-by: Kautuk Consul <kconsul@linux.vnet.ibm.com>
> > Tested-by: Hariharan T S <hariharan.ts@linux.vnet.ibm.com>
> > Message-Id: <20230327115030.3418323-2-kconsul@linux.vnet.ibm.com>
> > Message-Id: <20230330101141.30199-10-alex.bennee@linaro.org>
> >
> > ---
> > v2
> >    - limit --max-parallel-tasks $(JOBS_OPTION:-j%=%)
> > ---
> >   tests/Makefile.include | 18 +++++++++++-------
> >   tests/requirements.txt |  2 +-
> >   2 files changed, 12 insertions(+), 8 deletions(-)
> >
> > diff --git a/tests/Makefile.include b/tests/Makefile.include
> > index 9422ddaece..a4de0ad5a2 100644
> > --- a/tests/Makefile.include
> > +++ b/tests/Makefile.include
> > @@ -138,14 +138,18 @@ get-vm-image-fedora-31-%: check-venv
> >   # download all vm images, according to defined targets
> >   get-vm-images: check-venv $(patsubst %,get-vm-image-fedora-31-%, 
> > $(FEDORA_31_DOWNLOAD))
> >
> > +JOBS_OPTION=$(lastword -j1 $(filter-out -j, $(filter -j%,$(MAKEFLAGS))))
> > +
> >   check-avocado: check-venv $(TESTS_RESULTS_DIR) get-vm-images
> > -       $(call quiet-command, \
> > -            $(TESTS_PYTHON) -m avocado \
> > -            --show=$(AVOCADO_SHOW) run --job-results-dir=$(TESTS_RESULTS_DIR) \
> > -            $(if $(AVOCADO_TAGS),, --filter-by-tags-include-empty \
> > -                       --filter-by-tags-include-empty-key) \
> > -            $(AVOCADO_CMDLINE_TAGS) \
> > -            $(if $(GITLAB_CI),,--failfast) $(AVOCADO_TESTS), \
> > +       $(call quiet-command,                                                   
> > \
> > +            $(TESTS_PYTHON) -m avocado                                         
> >         \
> > +            --show=$(AVOCADO_SHOW) run --job-results-dir=$(TESTS_RESULTS_DIR)  
> > \
> > +            $(if $(AVOCADO_TAGS),,                                             
> > \
> > +                       --filter-by-tags-include-empty                          
> > \
> > +                       --filter-by-tags-include-empty-key)                     
> > \
> > +               --max-parallel-tasks $(JOBS_OPTION:-j%=%)                       
> > \
> > +            $(AVOCADO_CMDLINE_TAGS)                                            
> > \
> > +            $(if $(GITLAB_CI),,--failfast) $(AVOCADO_TESTS),                   
> > \
> >               "AVOCADO", "tests/avocado")
> >
> >   check-acceptance-deprecated-warning:
> > diff --git a/tests/requirements.txt b/tests/requirements.txt
> > index 0ba561b6bd..a6f73da681 100644
> > --- a/tests/requirements.txt
> > +++ b/tests/requirements.txt
> > @@ -2,5 +2,5 @@
> >   # in the tests/venv Python virtual environment. For more info,
> >   # refer to: https://pip.pypa.io/en/stable/user_guide/#id1
> >   # Note that qemu.git/python/ is always implicitly installed.
> > -avocado-framework==88.1
> > +avocado-framework==101.0
> >   pycdlib==1.11.0
> > --
> > 2.39.2
> I thought there were test failures that prohibited us from bumping the
> Avocado-Framework version. Did those get rectified recently?
Hi John,

While testing before and after behavior of pretty much every test in 
tests/avocado/,  I've found that some tests may get affected by the 
extra isolation (and overhead) of the new runner in recent Avocado and 
may get interrupted more often than on the previous runner.  But this is 
highly dependent on the amount of resources the machine you run the 
tests on has, and how tight the current timeout is.

Running all the tests under old and new Avocado also revealed some tests 
that are equally broken under both versions.  Thomas has bisected some, 
and Cedric has come up with some patches too.

To risk not being too verbose by default, let me know if you want the 
full details.  Also, see my previous reply about the two issues I'm 
aware that would make this bump as seamless as possible.

Thanks,

- Cleber.