Re: [RFC PATCH] tests/avocado: use new rootfs for orangepi test

[Philippe Mathieu-Daudé]

On 23/11/22 15:12, Alex Bennée wrote:
> Thomas Huth <thuth@redhat.com> writes:
> > On 23/11/2022 12.15, Philippe Mathieu-Daudé wrote:
> > > On 18/11/22 12:33, Alex Bennée wrote:
> > > > The old URL wasn't stable. I suspect the current URL will only be
> > > > stable for a few months so maybe we need another strategy for hosting
> > > > rootfs snapshots?
> > > >
> > > > Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
> > > > ---
> > > >    tests/avocado/boot_linux_console.py | 4 ++--
> > > >    1 file changed, 2 insertions(+), 2 deletions(-)
> > > >
> > > > diff --git a/tests/avocado/boot_linux_console.py
> > > > b/tests/avocado/boot_linux_console.py
> > > > index 4c9d551f47..5a2923c423 100644
> > > > --- a/tests/avocado/boot_linux_console.py
> > > > +++ b/tests/avocado/boot_linux_console.py
> > > > @@ -793,8 +793,8 @@ def test_arm_orangepi_sd(self):
> > > >            dtb_path =
> > > > '/usr/lib/linux-image-current-sunxi/sun8i-h3-orangepi-pc.dtb'
> > > >            dtb_path = self.extract_from_deb(deb_path, dtb_path)
> > > >            rootfs_url =
> > > > ('http://storage.kernelci.org/images/rootfs/buildroot/'
> > > > -                      'kci-2019.02/armel/base/rootfs.ext2.xz')
> > > > -        rootfs_hash = '692510cb625efda31640d1de0a8d60e26040f061'
> > > > +                      'buildroot-baseline/20221116.0/armel/rootfs.ext2.xz')
> > > > +        rootfs_hash = 'fae32f337c7b87547b10f42599acf109da8b6d9a'
> > > If Avocado doesn't find an artifact in its local cache, it will fetch it
> > > from the URL.
> > > The cache might be populated with artifacts previously downloaded, but
> > > their URL is not valid anymore (my case for many tests).
> > > We can also add artifacts manually, see [1].
> > > I'd rather keep pre-existing tests if possible, to test older
> > > (kernel / user-space) images. We don't need to run all the tests all
> > > the time:
> > > tests can be filtered by tags (see [2]).
> > > My preference here is to refactor this test, adding the
> > > "kci-2019.02"
> > > and "baseline-20221116.0" releases. I can prepare the patch if you /
> > > Thomas don't object.
> >
> > IMHO we shouldn't keep tests in the upstream git repository where the
> > binaries are not available in public anymore. They won't get run by
> > new contributors anymore, and also could vanish from the disks of the
> > people who previously downloaded it, once they wipe their cache or
> > upgrade to a new installation, so the test code will sooner or later
> > be bitrotting. But if you want to keep the tests around on your hard
> > disk, you could also stick the test in a local branch on your hard
> > disk instead.
>
> CI/Workstation splits aside I tend to agree with Thomas here that having
> tests no one else can run will lead to an accretion of broken tests.

Following this idea, should we remove all boards for which no open
source & GPL software is available? I.e:

40p                  IBM RS/6000 7020 (40p)
akita                Sharp SL-C1000 (Akita) PDA (PXA270)
midway               Calxeda Midway (ECX-2000)
terrier              Sharp SL-C3200 (Terrier) PDA (PXA270)
tosa                 Sharp SL-6000 (Tosa) PDA (PXA255)

> Given the tests themselves are standalone couldn't the prospective test
> hoarder keep their own personal repository to be run with the rest of the
> in-tree code, something like:
>
>    cd my/test/zoo/repo
>    $(QEMU_BUILD)/tests/venv/bin/avocado run my_test_zoo.py
>
> for convenience we could maybe support an env variable so the existing
> test selection tags would work:
>
>    set -x QEMU_AVOCADO_EXTRA_TESTS /my/test/zoo/repo
>    ./tests/venv/bin/avocado list
>    ...
>    <list all tests in qemu src tree and extra>
>    ...
>
> ?

Yes, this is what we use to test the Fuloong2E:

$ git grep RESCUE_YL_PATH tests/avocado/
tests/avocado/machine_mips_fuloong2e.py:21: 
@skipUnless(os.getenv('RESCUE_YL_PATH'), 'RESCUE_YL_PATH not available')
tests/avocado/machine_mips_fuloong2e.py:34:        kernel_path = 
self.fetch_asset('file://' + os.getenv('RESCUE_YL_PATH'),

The firmware is not open source / GPL but if you have a Fuloong2E board
you can dump it from the flash, then use it to test QEMU from hard reset
up to userland. Otherwise you are forced to use the -kernel argument.

> > The other possibility is to upload the binaries to a new public
> > location in the web ... but for software that contains GPLed software,
> > you should then also make sure to provide the source code to comply
> > with the license.
>
> This is the traditional reason we've lent so hard on external hosting
> for binaries because the upstream doesn't want the hassle of maintaining
> that sort of zoo of binaries. That said we have tests where binaries are
> served from fileserver.linaro.org but its then only my problem to deal
> with GPL requirements and not the upstream.

Maybe we are discussing 2 different topics. I am in possession of
old Solaris installation CDROMs and could boot some of them with
qemu-system-sparc. I want to automatize my testing, and wrote Avocado
scripts doing that. I suppose other QEMU users have similar CDROMs.
If I contribute my tests, they can run them. Isn't it in the interest
of the community to have such examples and tests available?

Regards,

Phil.