Re: [Qemu-devel] [PATCH v2 4/6] tests/acceptance: Add a BootLinuxConsoleMips test

[Philippe Mathieu-Daudé]

Hi Alex, Cleber.

On 06/28/2018 07:45 PM, Philippe Mathieu-Daudé wrote:
> On 06/28/2018 03:36 PM, Alex Bennée wrote:
>> Philippe Mathieu-Daudé <address@hidden> writes:
>>> On 06/28/2018 01:23 PM, Alex Bennée wrote:
>>>> Philippe Mathieu-Daudé <address@hidden> writes:
>>>>
>>>>> Similar to the BootLinuxConsoleX86_64 test:
>>>>> boot a Linux kernel on a Malta board and verify the serial is working.
>>>>>
>>>>> This test can be run using:
>>>>>
>>>>>     $ avocado run -t endian:big tests/acceptance
>>>>>
>>>>> Signed-off-by: Philippe Mathieu-Daudé <address@hidden>
>>>>> ---
>>>>>  tests/acceptance/boot_linux_console.py | 38 ++++++++++++++++++++++++++
>>>>>  1 file changed, 38 insertions(+)
>>>>>
>>>>> diff --git a/tests/acceptance/boot_linux_console.py 
>>>>> b/tests/acceptance/boot_linux_console.py
>>>>> index 17dc8d58c1..72cf5e943c 100644
>>>>> --- a/tests/acceptance/boot_linux_console.py
>>>>> +++ b/tests/acceptance/boot_linux_console.py
>>>>> @@ -46,3 +46,41 @@ class BootLinuxConsoleX86_64(Test):
>>>>>                  break
>>>>>              if 'Kernel panic - not syncing' in msg:
>>>>>                  self.fail("Kernel panic reached")
>>>>> +
>>>>> +
>>>>> +class BootLinuxConsoleMips(Test):
>>>>> +    """
>>>>> +    Boots a mips Linux kernel and checks that the console is operational
>>>>> +    and the kernel command line is properly passed from QEMU to the 
>>>>> kernel
>>>>> +
>>>>> +    :avocado: enable
>>>>> +    :avocado: tags=endian:big
>>>>> +    :avocado: tags=arch:mips
>>>>> +    :avocado: tags=board:malta
>>>>> +    """
>>>>> +
>>>>> +    arch = "mips"
>>>>> +    timeout = 60
>>>>> +
>>>>> +    def test(self):
>>>>> +        kernel_url = ('http://people.debian.org/~aurel32/qemu/mips/'
>>>>> +                      'vmlinux-3.2.0-4-4kc-malta')
>>>>> +        kernel_hash = '592e384a4edc16dade52a6cd5c785c637bcbc9ad'
>>>>> +        kernel_path = self.fetch_asset(kernel_url,
>>>>> asset_hash=kernel_hash)
>>>>
>>>> I'm uncomfortable using "random" binaries of websites as the source of
>>>> our test kernels. I can see the justification for distro kernels as they
>>>> at least have the infrastructure to rebuild from source if you really
>>>> want to, but even then the distros don't cover a lot of the
>>>> architectures.

Alex: I could find all the Linux kernel I'm interested to console-test
with Avocado on the http://snapshot.debian.org/ archive website.

For example Aurelien's one (more up-to-date) is available here:
http://snapshot.debian.org/package/linux-2.6/2.6.32-48/#linux-image-2.6.32-5-4kc-malta_2.6.32-48

I also added a SH-4 test for the SM501 series of Zoltan BALATON using
the kernel extracted from this distrib built kernel:
http://snapshot.debian.org/package/linux-2.6/2.6.32-30/#linux-image-2.6.32-5-sh7751r_2.6.32-30

The Debian distribution also provide the source package and the kernels
can be simply rebuilt using make-kpkg or (make bindeb-pkg with more
recent kernels).

Would it be enough to satisfy the GPL requirements to provided that info
in the header and use these handy pre-compiled kernels?

Cleber: As you added tarballs and zip support, I'd also need a "dpkg"
support in your fetch_asset() archive management...

Cleber: And another future request is to be able to extract files from
ISO images with the guestfish tool :P
To be able to add a test for the Apple Machintosh Quadra 800 machine:
http://lists.nongnu.org/archive/html/qemu-devel/2018-06/msg08139.html

>>> And now I notice I made an mistake here :) I guess remember the Avocado
>>> team started using the SHA-1 hash as default and I suggested them to be
>>> able to use other hashes for this particular case, since Aurelien
>>> provided the MD5 hashes signed by his GPG key, which is signed/trusted
>>> by Peter and used to merge mips32 pulls.
>>>
>>> That would verify the QEMU community circle of trust right?
>>
>> It's not so much the chain of trust and more repeatability of building
>> the test cases. I trust Aurel32's binaries are good test cases for MIPS
>> but at the moment I have no idea how to rebuild them which is a bit of
>> an issue given it is covered by the GPL.
> 
> OK! I didn't even think about that since this is a "Linux" kernel and a
> "Debian" rootfs provided by a Debian developer, hosted on a Debian
> server, so I'm pretty sure this is GPL compliant, but it makes sens.
> 
> I'll find a way to have reproducible test binaries for acceptance testing.
> 
>>> I don't think Avocado should parse the FTP/HTTP signed indexes, but a
>>> manual verification when merging this series should suffice.
>>>
>>>>
>>>> I had experimented with using docker based builds for building test
>>>> fixtures (see tests/docker/dockerbuilds):
>>>>
>>>>    https://github.com/stsquad/qemu/tree/docker/linux-user-and-ltp-builds-v2
>>>>
>>>> As these tests are fairly simple boot tests that just need kernels maybe
>>>> we could look at tooling up the generation of these images in a
>>>> repeatable way - similar to the way vmtest builds VMs.
>>>
>>> Yes, I have another acceptance branch where I cross-build an old mipssim
>>> kernel to test the board, using the following:
>>> http://lists.nongnu.org/archive/html/qemu-devel/2018-04/msg04071.html
>>>
>>> But preparing a Docker cross image, fetching the Linux kernel source,
>>> building it, takes a lot of time/storage I'd rather avoid; at least with
>>> Aurelien kernels, since they are known to work since years.
>>
>> Well you can throw away the docker image as long as we have a mechanism
>> to dump out the final binary. I have no interest in forcing people to
>> keep checked out code using up space, just that they can re-build if
>> they need to.
> 
> The Avocado guys said they have plenty of storage for assets.
> A cool feature would be to have some config built by some CI builder
> that then sign and push reproducible config + generated binary to the
> assets storage automatically.
> 
>>>>> +
>>>>> +        self.vm.set_machine('malta')
>>>>> +        self.vm.set_console()
>>>>> +        kernel_command_line = 'console=ttyS0 printk.time=0'
>>>>> +        self.vm.add_args('-m', "64",
>>>>> +                         '-kernel', kernel_path,
>>>>> +                         '-append', kernel_command_line)
>>>>> +        self.vm.launch()
>>>>> +        console = self.vm.console_socket.makefile()
>>>>> +        console_logger = logging.getLogger('console')
>>>>> +        while True:
>>>>> +            msg = console.readline()
>>>>> +            console_logger.debug(msg.strip())
>>>>> +            if 'Kernel command line: %s' % kernel_command_line in msg:
>>>>> +                break
>>>>> +            if 'Kernel panic - not syncing' in msg:
>>>>> +                self.fail("Kernel panic reached")
>>>>
>>>> Of course for bonus points a simple rootfs with hackbench or some such
>>>> in it would be nice. But I appreciate this makes the building job a lot
>>>> more complex than just a kernel.
>>>
>>> My idea is to use the rootfs for larger tests, and tag the "Kernel
>>> panic" tests as "quick", so we can have a "make acceptance-speed" or
>>> similar.
>>>
>>> We can already test than many devices were initialized correctly quickly
>>> looking at this console output.
>>
>> Yeah the kernel boot up is a pretty good smoke test (it's all most of
>> kernelci manages as well). However it certainly doesn't fully exercise
>> the translator. I've lost track of the number of bugs we found after
>> successfully booting an kernel because we were doing exhaustive
>> instruction testing.
> 
> This is not indeed a translator test, but simple enough to test than
> devices are instantiated, mapped at the correct address, detected by
> Linux, probe the device enough to initialize it.
> 
> I actually wrote this test after breaking a I/O device why a previous
> Super I/O refactor, this test would have failed.
> 
> You could say a simple "info mtree" parser would notice it too... but
> acceptance testing is closer to real-world imho, and would be also
> useful to catch changes in the guest code (as in "QEMU expects things
> that way, but the Linux kernel updated to something new").
> 
> Regards,
> 
> Phil.
>

signature.asc
Description: OpenPGP digital signature