[Qemu-devel] [PATCH 0/6] QTests: use Python to run complex tests

[Philippe Mathieu-Daudé]

Hi,

With this series we can now write tests using Python rather than C.
For complex tests this can reduce the test development time, we can focus on
the test purposes instead of his implementation details.

- 1,2: we already have Python classes to run Block tests, move all the
  non Block-related methods to qtest.py,
- 3: default TEST_DIR to TMPDIR,
- 4: add a method to restrict tests to a list of supported machines,
- 5: since the Block tests are output sensitive, do not mess with their
  current tuned iotests::main(unittest), add a more generic one in qtest.py,
- 6: finally modify the tests Makefile to run C/Python tests with the same
  rule.

to have a better idea, here is a snippet from the next series:

    class TestSdCardSpecV2(qtest.QMPTestCase):
        [...]
        def test_power_on_spec_v2(self):
            self.bus.do_cmd(GO_IDLE_STATE)
            [...]
            # verify Card ID
            data = self.bus.do_cmd(ALL_SEND_CID)
            oid, pnm, psn = struct.unpack(">x2s5sxLxxx", data)
            self.assertEqual(oid, "XY") # QEMU default
            self.assertEqual(pnm, "QEMU!") # QEMU default
            self.assertEqual(psn, 0xdeadbeef) # QEMU default

The test suite is ran the same way:

$ make check-qtest-aarch64 
  GTESTER check-qtest-aarch64
  PYTEST  check-qtest-aarch64

Adding the verbose flag we see the SD bus commands:

$ make check-qtest-aarch64 V=1
TEST: tests/test-hmp... (pid=16074)
...
  /aarch64/hmp/xlnx-zcu102:                                            OK
  /aarch64/hmp/lm3s811evb:                                             OK
  /aarch64/hmp/none+2MB:                                               OK
PASS: tests/test-hmp
PYTHONPATH=/source/qemu/scripts QEMU_PROG=aarch64-softmmu/qemu-system-aarch64 
TEST_DIR=/tmp python -B /source/qemu/tests/sdcard_tests.py -v
INFO:root:CMD#00 -> (none)
INFO:root:CMD#08 -> 00000100
INFO:root:CMD#08 -> 00000200
INFO:root:CMD#08 -> 00000400
INFO:root:CMD#08 -> 00000800
INFO:root:CMD#08 -> 00001000
INFO:root:CMD#08 -> 00002000
INFO:root:CMD#55 -> 00000120
INFO:root:CMD#41 -> 00ffff00
INFO:root:CMD#55 -> 00000120
INFO:root:CMD#41 -> 80ffff00
INFO:root:CMD#02 -> aa585951454d552101deadbeef006219
INFO:root:CMD#03 -> 45670500
INFO:root:CMD#03 -> 8ace0700
.
----------------------------------------------------------------------
Ran 1 test in 0.070s

OK

Regards,

Phil.

Based-on: address@hidden
          (QOM'ify SDBus, housekeeping)

Philippe Mathieu-Daudé (6):
  iotests.py: split BlockQMPTestCase class of QMPTestCase
  iotests.py: move the generic QMPTestCase to qtest.py
  qtest.py: use TMPDIR/TEMP if the TEST_DIR env var is missing
  qtest.py: add verify_machine(supported_machines)
  qtest.py: add a simple main() which calls unittest.main()
  tests: add a Makefile rule to run Python qtests

 scripts/qtest.py              | 152 ++++++++++++++++++++++++++++++++++++++++++
 tests/Makefile.include        |  11 ++-
 tests/qemu-iotests/030        |  14 ++--
 tests/qemu-iotests/040        |   2 +-
 tests/qemu-iotests/041        |  20 +++---
 tests/qemu-iotests/044        |   2 +-
 tests/qemu-iotests/045        |   4 +-
 tests/qemu-iotests/055        |   8 +--
 tests/qemu-iotests/056        |   4 +-
 tests/qemu-iotests/057        |   2 +-
 tests/qemu-iotests/065        |   2 +-
 tests/qemu-iotests/093        |   6 +-
 tests/qemu-iotests/096        |   2 +-
 tests/qemu-iotests/118        |   2 +-
 tests/qemu-iotests/124        |   2 +-
 tests/qemu-iotests/129        |   2 +-
 tests/qemu-iotests/132        |   2 +-
 tests/qemu-iotests/136        |   2 +-
 tests/qemu-iotests/139        |   2 +-
 tests/qemu-iotests/147        |   2 +-
 tests/qemu-iotests/148        |   2 +-
 tests/qemu-iotests/152        |   2 +-
 tests/qemu-iotests/155        |   2 +-
 tests/qemu-iotests/163        |   2 +-
 tests/qemu-iotests/165        |   2 +-
 tests/qemu-iotests/196        |   2 +-
 tests/qemu-iotests/iotests.py |  84 ++---------------------
 27 files changed, 212 insertions(+), 127 deletions(-)

-- 
2.15.1