[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [Qemu-devel] [RFC PATCH 0/7] avocado: Add acceptance tests parsing t
From: |
Philippe Mathieu-Daudé |
Subject: |
Re: [Qemu-devel] [RFC PATCH 0/7] avocado: Add acceptance tests parsing the Linux boot console |
Date: |
Fri, 20 Apr 2018 09:59:43 -0300 |
User-agent: |
Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.7.0 |
> Cross-posting qemu-devel + avocado-devel.
>
>> While previously working on a Super I/O refactor, I encountered some problems
>> at runtime, after building the codebase successfully and running qtests.
>> I had to manually start to boot different guests and check the bootlog.
>>
>> I wanted to give a try at Avocado which seems designed to simplify that kind
>> of functional tests.
>>
>> I applied Amador Pahim work following Cleber Rosa documentation from
>> http://lists.nongnu.org/archive/html/qemu-devel/2018-01/msg03891.html,
>> however I had to modify few things to parse the boot console.
>> Since his work is not merged, I included it in this series.
>>
>> The tests simply expect to find a string reported by Linux printk when a
>> device is detected/initialized, such "ttyS0 at I/O 0x3f8 (irq = 4) is a
>> 16550A"
>> and "i8042 KBD port at 0x60,0x64 irq 1" for the Super I/O chip, or such
>> "registered as PCnet/PCI II 79C970A" to confirms the PCI subsystem and
>> network
>> device are correctly detected:
>>
>>
>> self.assertIn(u'ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A', bootlog)
>> self.assertIn(u'ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A', bootlog)
>> self.assertIn(u'i8042 KBD port at 0x60,0x64 irq 1', bootlog)
>> self.assertIn(u'i8042 AUX port at 0x60,0x64 irq 12', bootlog)
>>
>> Example of the tests output:
>>
>> $ avocado run test_linux-boot-console.py -m
>> test_linux-boot-console.py.data/parameters.yaml
>> JOB ID : 695094c9bbe8f6011226da7c2031c2c53e949910
>> JOB LOG :
>> /home/phil/avocado/job-results/job-2018-04-19T13.36-695094c/job.log
>> (1/6)
>> test_linux-boot-console.py:TestAlphaClipperBoot2_6.test_boot_console;alpha-2582:
>> PASS (4.76 s)
>> (2/6)
>> test_linux-boot-console.py:TestAlphaClipperBoot2_6.test_boot_console;mips-4a72:
>> PASS (0.00 s)
>> (3/6)
>> test_linux-boot-console.py:TestMips4kcMaltaBoot2_6.test_boot_console;alpha-2582:
>> PASS (0.00 s)
>> (4/6)
>> test_linux-boot-console.py:TestMips4kcMaltaBoot2_6.test_boot_console;mips-4a72:
>> PASS (3.92 s)
>> (5/6)
>> test_linux-boot-console.py:TestMips4kcMaltaBoot3_2.test_boot_console;alpha-2582:
>> PASS (0.00 s)
>> (6/6)
>> test_linux-boot-console.py:TestMips4kcMaltaBoot3_2.test_boot_console;mips-4a72:
>> PASS (4.08 s)
>> RESULTS : PASS 6 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 |
>> CANCEL 0
>> JOB TIME : 13.31 s
>>
>> Please apologize my ugly Python, this series is just a proof-of-concept :)
>> I couldn't figure out how to use the @skipUnless(correct arch) decorator.
>
> Eduardo asked me to share my first impressions after trying this
> framework with QEMU.
>
> So far it seems more designed to run tests _inside_ a qemu instance
> (think user-space), eventually testing packages/scripts.
Current examples use the same pattern, which I felt a bit restricting:
[QemuTest]
.setUp: prepare VM arguments, launch the VM
.test_*: run tests in the running VM
.tearDown: shutdown the VM and cleanup dangling files
> My tests are targeting the machine model itself, if the devices are
> correctly instantiated and so.
> The fastest approach was to check the Linux kernel bootlog, but if you
> look at the TestAlphaClipperBoot console you'll see a boot "firmware" is
> executed before the kernel. Casually this firmware also send information
> on the console, but what if the console is not accessible?
> We can use a chardev for the ioport80 POST, but some firmwares post boot
> events via I2C, SPI, CAN...
>
> To improve the testing, my idea is to use the Tracing framework.
> The machine trace events would be logged in a db and avogado would
> verify some of the trace events, did they occur? in the correct sequence
> order? with the correct arguments? We can also check relative
> timestamps, is this sequence timing fast enough?
>
> I plan to use the gdb API to insert breakpoints and follow part
> execution flow, eventually injecting (faulty) events.
>
> The multi-arch/machines support is a bit weak yet.
>
> I'm personally more interested in automatically testing [real world]
> firmware or full machine boot process, the kind of integration testing I
> can not do with qtests.
>
> I liked the possibility to generate coredumps, or the replay function.
>
> I also liked to be able to write a test on how a machine boots, in ~20
> LOC (see TestAlphaClipperBoot2_6).
>
> The storage API is OK to fetch a full VM image, but not a single file
> like a kernel or a flash image
signature.asc
Description: OpenPGP digital signature
- [Qemu-devel] [RFC PATCH 3/7] qemu.py: Check console arch is supported before calling mktemp(), (continued)
- [Qemu-devel] [RFC PATCH 3/7] qemu.py: Check console arch is supported before calling mktemp(), Philippe Mathieu-Daudé, 2018/04/19
- [Qemu-devel] [RFC PATCH 2/7] avocado: Update python scripts to upstream codebase, Philippe Mathieu-Daudé, 2018/04/19
- [Qemu-devel] [RFC PATCH 1/7] AVOCADO_QEMU: Snapshot commit, Philippe Mathieu-Daudé, 2018/04/19
- [Qemu-devel] [RFC PATCH 4/7] qemu.py: Avoid deprecated tempfile.mktemp(), Philippe Mathieu-Daudé, 2018/04/19
- [Qemu-devel] [RFC PATCH 5/7] avocado: Add an optional flag 'login' to get_console(), Philippe Mathieu-Daudé, 2018/04/19
- [Qemu-devel] [RFC PATCH 6/7] avocado: Add a test parsing Linux kernel booting console, Philippe Mathieu-Daudé, 2018/04/19
- [Qemu-devel] [RFC PATCH 7/7] avocado: Add tests booting the Malta machine, Philippe Mathieu-Daudé, 2018/04/19
- Re: [Qemu-devel] [RFC PATCH 0/7] avocado: Add acceptance tests parsing the Linux boot console, no-reply, 2018/04/19
- Re: [Qemu-devel] [RFC PATCH 0/7] avocado: Add acceptance tests parsing the Linux boot console, Philippe Mathieu-Daudé, 2018/04/20
- Re: [Qemu-devel] [RFC PATCH 0/7] avocado: Add acceptance tests parsing the Linux boot console,
Philippe Mathieu-Daudé <=
- Re: [Qemu-devel] [RFC PATCH 0/7] avocado: Add acceptance tests parsing the Linux boot console, Cleber Rosa, 2018/04/30