Re: [PATCH 1/5] tests/boot_linux_console: Add initrd test for the Exynos4210

[Philippe Mathieu-Daudé]

On 10/9/19 9:07 PM, Cleber Rosa wrote:
> On Wed, Oct 09, 2019 at 02:38:02PM +0100, Peter Maydell wrote:
> > On Tue, 8 Oct 2019 at 22:49, Cleber Rosa <address@hidden> wrote:
> > > On Mon, Oct 07, 2019 at 05:28:49PM +0100, Peter Maydell wrote:
>>>
>>>> Do our other acceptance tests download random third-party
>>>> (ie "not a well-known distro") binaries for the tests ?
>>>> It seems a bit hazardous for reproducability and trustability
>>>> reasons...
[...]
> > > I find it hard to judge precisely how much of a third-party some of
> > > these are.  I remember Philippe mentioning that one of them, I guess
> > > the images used on linux_ssh_mips_malta.py, were "as official as it
> > > gets" (my words, from my often misleading memory).
> > >
> > > Reproducibility is definitely an issue, in the sense given that some
> > > of these can indeed go away, but as long as they're available the hash
> > > recorded in the test should guarantee that we're running the same
> > > images.

So this thread is a follow up on:
https://www.mail-archive.com/address@hidden/msg546430.html

For Open Source software I can understand we want to be able to rebuild 
them, and should provide a link about how to rebuild.

I don't want to rebuild images myself, I want to focus on testing QEMU. 
I tried once to build a MIPSsim kernel and it was a total nightmare:
https://lists.nongnu.org/archive/html/qemu-devel/2018-04/msg04071.html
(Thomas Huth eventually succeeded using buildroot).

> > > Do you think we should do something different here?
> >
> > I'm not sure, which is why I asked whether this new test
> > was in line with what we've done previously. Since these
> > are just test cases and we don't redistribute them to
> > other people there's less of a traceability/reproducibility
> > worry, and if we check hashes on download that cuts off
> > a lot of "fail to notice if the image changes for some
> > reason" possible problems.
>
> Yep, because I have no clue how to do improve on this (redistributing
> the binaries is definitely not on the improvement side, and neither
> is not testing some machine types), the current approach seems good.

QEMU machines are not restricted to running Linux or other Open Source 
software. It seems important to be able to test for regressions with 
closed-source code too, because it usually has been well tested on real 
hardware.

I just posted a firmware test:
https://www.mail-archive.com/address@hidden/msg651012.html

I could find it stored compressed and uuencoded on the Wayback Machine.
Since I don't want to abuse from this amazing service, and other script 
to decode/uncompress it, I stored it on a new repository with its SHA-1
(the default hash used by Avocado):
https://github.com/philmd/qemu-testing-blob/tree/master/hppa/hp9000/712

Is this acceptable? Incorrect?

Regarding Avocado tests, maybe we can simply add a "untrusted" or 
"closedsource" tag, so people willing to test untrusted binaries could 
still run the tests using 'avocado --tag untrusted_software', and we 
could skip these tests by default.

I am very interested because I already experimented with:

- AIX on PReP/40p
- some RTOS on Canon PowerShot A1100
- VxWorks on MIPS and SPARC
- closed-source bootloader for Raspi3/4

Regards,

Phil.