Re: [Qemu-devel] [PATCH for-4.0 0/2] test: replace gtester with a TAP driver

[Thomas Huth]

On 2018-11-30 15:47, Cleber Rosa wrote:
> 
> 
> On 11/30/18 4:58 AM, Paolo Bonzini wrote:
>> On 30/11/18 08:21, Thomas Huth wrote:
>>> On 2018-11-29 21:43, Eric Blake wrote:
>>>> On 11/29/18 11:45 AM, Paolo Bonzini wrote:
>>>>> gtester is deprecated by upstream glib and it does not support tests
>>>>> that call g_test_skip in some glib stable releases.
>>>>>
>>>>> glib suggests instead using Automake's TAP support.  We do not support
>>>>> Automake, but we can copy the code that beautifies the TAP output and
>>>>> use it.  I chose to use the Perl copy rather than the shell/awk one,
>>>>> in order to reuse Perl's TAP parsing package, but I'm open to suggestions
>>>>> about which language to use.
>>>>
>>>> I'm less familiar with the TAP protocol than I'd like to admit, but I
>>>> did find:
>>>>
>>>> https://blog.gtk.org/2018/07/11/news-from-glib-2-58/
>>>>
>>>> which corroborates your claim that switching to Automake's TAP parser is
>>>> indeed a recommended approach to avoid the now-deprecated gtester.
>>>
>>> Hmm, do we really have to maintain our own version of a test runner now?
>>
>> Well, that's what you get for using a custom build system but it's
>> "just" an output beautifier really.  I could just "grep -ve ^# -e ^ok"
>> but it wouldn't look very nice (plus things that start as 1-line shell
>> scripts end up being the same as the thing you're trying to replace).
>>
>> I don't think it will be a big maintenance headache---just like most
>> other files in scripts/, some of which have had exactly zero commits
>> since they were added.  In fact, Automake has hardly changed its driver
>> since the first commit.

OK, fair.

>>> There's already some effort going on to use avocado in "make
>>> check-acceptance" ... maybe we could use avocado as replacement for
>>> gtester, too?
>>
>> Avocado does not have the required functionality, unfortunately.  It can
>> produce TAP, but not consume it.  Also, Python libraries for TAP do not
>> abound and are generally not shipped with the OS (apart from Perl, which
>> basically invented TAP, the language with the most TAP libraries seem to
>> be JavaScript!).
> 
> Avocado can't currently consume TAP (it produces it, though), but it has
> some integration with glib based tests:
> 
> https://avocado-framework.readthedocs.io/en/66.0/optional_plugins/glib.html
> 
> But, there may be bits missing to fully replae gtests.  Given that we
> have been adding features to Avocado based on QEMU requirements, we can
> certainly look at adding missing pieces there.  If this sounds like
> something you'd be interested in, I can come up with an integration
> proposal.

FWIW, it seems basically to work, indeed:

$ export QTEST_QEMU_BINARY=x86_64-softmmu/qemu-system-x86_64
$ avocado run --loaders glib -- tests/pxe-test tests/cdrom-test
JOB ID     : 67657631c2f1f80e34d90fe0c2c05e05aab6f1ba
JOB LOG    : 
/home/thuth/avocado/job-results/job-2018-11-30T10.02-6765763/job.log
 (01/12) tests/pxe-test:/x86_64/pxe/ipv4/pc/e1000: PASS (6.46 s)
 (02/12) tests/pxe-test:/x86_64/pxe/ipv4/pc/virtio-net-pci: PASS (6.47 s)
 (03/12) tests/pxe-test:/x86_64/pxe/ipv4/q35/e1000e: PASS (6.46 s)
 (04/12) tests/pxe-test:/x86_64/pxe/ipv4/q35/virtio-net-pci: PASS (6.46 s)
 (05/12) tests/cdrom-test:/x86_64/cdrom/boot/default: PASS (0.25 s)
 (06/12) tests/cdrom-test:/x86_64/cdrom/boot/virtio-scsi: PASS (0.25 s)
 (07/12) tests/cdrom-test:/x86_64/cdrom/boot/isapc: PASS (0.24 s)
 (08/12) tests/cdrom-test:/x86_64/cdrom/boot/am53c974: PASS (0.25 s)
 (09/12) tests/cdrom-test:/x86_64/cdrom/boot/dc390: PASS (0.25 s)
 (10/12) tests/cdrom-test:/x86_64/cdrom/boot/lsi53c895a: PASS (0.26 s)
 (11/12) tests/cdrom-test:/x86_64/cdrom/boot/megasas: PASS (0.15 s)
 (12/12) tests/cdrom-test:/x86_64/cdrom/boot/megasas-gen2: PASS (0.15 s)
RESULTS    : PASS 12 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | 
CANCEL 0
JOB TIME   : 28.31 s
JOB HTML   : 
/home/thuth/avocado/job-results/job-2018-11-30T10.02-6765763/results.html

 Thomas