Re: [Qemu-devel] [PATCH v2 0/6] Avocado: start multi-arch tests, add a Travis job

[Alex Bennée]

Philippe Mathieu-Daudé <address@hidden> writes:

> Hi Alex,
>
> On 06/28/2018 01:01 PM, Alex Bennée wrote:
>> Philippe Mathieu-Daudé <address@hidden> writes:
>>>
>>> Another neanderthal approach to add multi-arch acceptance tests using
>>> Avocado.
>>>
>>> Since Cleber Rosa work got merged [0], I can restart my previous attempt [1]
>>> at using Avocado in QEMU.
>>> Cleber Rosa rewrote my previous v1 [1], in good python, now this v2 pushes
>>> a bit further:
>>> - we can now run other archs
>>> - Avocado is integrated into Travis CI
>>>
>>> I'm not sure Travis is the best CI available for those tests, but it runs
>>> quite fast:
>>>
>>>       ...
>>>       LINK    x86_64-softmmu/qemu-system-x86_64
>>>     JOB ID     : a3a56af3c02d193e862ce660aae1d9c72926dcb6
>>>     JOB LOG    : 
>>> /home/travis/avocado/job-results/job-2018-06-21T23.49-a3a56af/job.log
>>>      (1/7) 
>>> tests/acceptance/boot_linux_console.py:BootLinuxConsoleX86_64.test:  PASS 
>>> (3.89 s)
>>>      (2/7) 
>>> tests/acceptance/boot_linux_console.py:BootLinuxConsoleMips.test:  PASS 
>>> (1.83 s)
>>>      (3/7) tests/acceptance/version.py:Version.test_qmp_human_info_version: 
>>>  PASS (0.05 s)
>>>      (4/7) tests/acceptance/vnc.py:Vnc.test_no_vnc:  PASS (0.04 s)
>>>      (5/7) tests/acceptance/vnc.py:Vnc.test_no_vnc_change_password:  PASS 
>>> (0.04 s)
>>>      (6/7) 
>>> tests/acceptance/vnc.py:Vnc.test_vnc_change_password_requires_a_password:  
>>> PASS (0.05 s)
>>>      (7/7) tests/acceptance/vnc.py:Vnc.test_vnc_change_password:  PASS 
>>> (0.05 s)
>>>     RESULTS    : PASS 7 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 |
>>>       INTERRUPT 0 | CANCEL 0
>>
>> That said on my AArch64 server it fairs better:
>>
>> 15:59:19 address@hidden:~/l/qemu.git] review/avocado-v2 + avocado run 
>> tests/acceptance
>> JOB ID     : f40e814204ff007c58073ca75abb38bd575f39f2
>> JOB LOG    : 
>> /home/alex.bennee/avocado/job-results/job-2018-06-28T15.59-f40e814/job.log
>>  (1/7) tests/acceptance/boot_linux_console.py:BootLinuxConsoleX86_64.test: 
>> CANCEL (0.00 s)
>>  (2/7) tests/acceptance/boot_linux_console.py:BootLinuxConsoleMips.test: 
>> PASS (2.03 s)
>>  (3/7) tests/acceptance/version.py:Version.test_qmp_human_info_version: 
>> CANCEL (0.00 s)
>>  (4/7) tests/acceptance/vnc.py:Vnc.test_no_vnc: CANCEL (0.00 s)
>>  (5/7) tests/acceptance/vnc.py:Vnc.test_no_vnc_change_password: CANCEL (0.00 
>> s)
>>  (6/7) 
>> tests/acceptance/vnc.py:Vnc.test_vnc_change_password_requires_a_password: 
>> CANCEL (0.00 s)
>>  (7/7) tests/acceptance/vnc.py:Vnc.test_vnc_change_password: CANCEL (0.00 s)
>> RESULTS    : PASS 1 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | 
>> CANCEL 6
>> JOB TIME   : 2.47 s
>>
>> Although given the errors in the logs I'm surprised at ERROR 0
>
> Why are you surprised? You were expecting the BootLinuxConsoleMips to
> fail?

No but a bunch have CANCELed... maybe they are skipping but if the test
is run and it fails to complete I would expect that to be an error.

>
>>
>>>     JOB TIME   : 6.21 s
>>>
>>>     Done. Your build exited with 0.
>>>
>>>     Job ran for 5 min 34 sec
>>>
>>> I couldn't add the Alpha tests because all the vmlinux kernels I found
>>> online are gzipped, so I'm waiting some upstream Avocado merges before.
>>>
>>> Cleber: I hope you can help me pythonizing this series :p
>>>
>>> Regards,
>>>
>>> Phil.
>>>
>>> [0]: http://lists.nongnu.org/archive/html/qemu-devel/2018-06/msg04531.html
>>> [1]: http://lists.nongnu.org/archive/html/qemu-devel/2018-04/msg03076.html
>>>
>>> Philippe Mathieu-Daudé (6):
>>>   avocado: Add a Test.arch property
>>>   tests/acceptance: Rename the x86-64 specific BootLinuxConsole test
>>>   tests/acceptance: Improve the Avocado tags
>>>   tests/acceptance: Add a BootLinuxConsoleMips test
>>>   tests/acceptance: Add a kludge to not use the default console
>>>   travis: Add Avocado tests
>>>
>>>  .travis.yml                               |  8 +++++
>>>  scripts/qemu.py                           |  6 ++--
>>>  tests/acceptance/avocado_qemu/__init__.py | 17 ++++++---
>>>  tests/acceptance/boot_linux_console.py    | 44 +++++++++++++++++++++--
>>>  4 files changed, 67 insertions(+), 8 deletions(-)
>>
>>
>> --
>> Alex Bennée
>>


--
Alex Bennée