Re: [Qemu-block] Failing iotests in CI (was: Add a gitlab-ci file for Continuous Integration testing on Gitlab)

[Thomas Huth]

On 19/02/2019 10.37, Kevin Wolf wrote:
> Am 19.02.2019 um 10:04 hat Thomas Huth geschrieben:
>> On 19/02/2019 08.53, Kevin Wolf wrote:
[...]
>>> Which are the cases that fail for you with '--disable-tcg'?
>>
>> These tests are failing: 087 169 188 232 235 238
> 
> Hm, 087 and 232 just do something like:
> 
>     $QEMU_PROG -nodefaults -machine accel=qtest -nographic \
>                -qmp stdio -serial none \
>                ...some -drive and -object options...
> 
> This should be fine with --disable-tcg, I think?
> 
> 169 runs a VM, but I don't see anything that makes it use TCG.
> 
> 188 doesn't even run QEMU at all, it's only qemu-io. I don't see how
> this could be possibly related to --disable-tcg.

087 and 188 obviously simply lack a check for the required crypto support.


Output of 087:

087                [08:33:55] [08:33:56] - output mismatch (see 087.out.bad)
--- /builds/huth/qemu/tests/qemu-iotests/087.out        2019-02-19 
08:23:54.000000000 +0000
+++ /builds/huth/qemu/tests/qemu-iotests/087.out.bad    2019-02-19 
08:33:56.000000000 +0000
@@ -46,12 +46,13 @@
 
 === Encrypted image LUKS ===
 
+qemu-img: TEST_DIR/t.IMGFMT: No crypto library supporting PBKDF in this build: 
Function not implemented
 Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=134217728 encrypt.format=luks 
encrypt.key-secret=sec0
 Testing:
 QMP_VERSION
 {"return": {}}
 {"return": {}}
-{"return": {}}
+{"error": {"class": "GenericError", "desc": "No encryption in image header, 
but options specified format 'luks'"}}
 {"return": {}}
 {"timestamp": {"seconds":  TIMESTAMP, "microseconds":  TIMESTAMP}, "event": 
"SHUTDOWN", "data": {"guest": false, "reason": "host-qmp-quit"}}


And the output of 188:

188                [08:34:53] [08:34:54] - output mismatch (see 188.out.bad)
--- /builds/huth/qemu/tests/qemu-iotests/188.out        2019-02-19 
08:23:54.000000000 +0000
+++ /builds/huth/qemu/tests/qemu-iotests/188.out.bad    2019-02-19 
08:34:54.000000000 +0000
@@ -1,4 +1,5 @@
 QA output created by 188
+qemu-img: TEST_DIR/t.IMGFMT: No crypto library supporting PBKDF in this build: 
Function not implemented
 Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=16777216 encrypt.format=luks 
encrypt.key-secret=sec0 encrypt.iter-time=10
 
 == reading whole image ==
@@ -14,5 +15,6 @@
 16 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
 
 == verify open failure with wrong password ==
-can't open: Invalid password, cannot unlock any keyslot
+read 16777216/16777216 bytes at offset 0
+16 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
 *** done


169 got killed via abort():

169                [08:34:39] [08:34:46] [failed, exit status 1] - output 
mismatch (see 169.out.bad)
--- /builds/huth/qemu/tests/qemu-iotests/169.out        2019-02-19 
08:23:54.000000000 +0000
+++ /builds/huth/qemu/tests/qemu-iotests/169.out.bad    2019-02-19 
08:34:46.000000000 +0000
@@ -1,5 +1,29 @@
-....................
+WARNING:qemu:qemu received signal 6: 
/builds/huth/qemu/tests/qemu-iotests/../../x86_64-softmmu/qemu-system-x86_64 
-chardev 
socket,id=mon,path=/tmp/qemu-iotests-quick-25045/tmpGQOExQ/qemua-13044-monitor.sock
 -mon chardev=mon,mode=control -display none -vga none -qtest 
unix:path=/tmp/qemu-iotests-quick-25045/qemua-13044-qtest.sock -machine 
accel=qtest -nodefaults -machine accel=qtest -drive 
if=virtio,id=drive0,file=/tmp/qemu-iotests-quick-25045/disk_a,format=qcow2,cache=writeback
[...]

No clue why.


232 is also strange, no idea what is going on here:

232                [08:38:53] [08:38:56] - output mismatch (see 232.out.bad)
--- /builds/huth/qemu/tests/qemu-iotests/232.out        2019-02-19 
08:23:54.000000000 +0000
+++ /builds/huth/qemu/tests/qemu-iotests/232.out.bad    2019-02-19 
08:38:56.000000000 +0000
@@ -21,13 +21,13 @@
 NODE_NAME: TEST_DIR/t.IMGFMT (file, read-only)
 NODE_NAME: TEST_DIR/t.IMGFMT (file, read-only)
 
-QEMU_PROG: -drive 
driver=file,file=TEST_DIR/t.IMGFMT,if=none,read-only=off,auto-read-only=off: 
Could not open 'TEST_DIR/t.IMGFMT': Permission denied
-NODE_NAME: TEST_DIR/t.IMGFMT (file, read-only)
-NODE_NAME: TEST_DIR/t.IMGFMT (file, read-only)
-
-QEMU_PROG: -drive 
driver=file,file=TEST_DIR/t.IMGFMT,if=none,auto-read-only=off: Could not open 
'TEST_DIR/t.IMGFMT': Permission denied
-NODE_NAME: TEST_DIR/t.IMGFMT (file, read-only)
-NODE_NAME: TEST_DIR/t.IMGFMT (file, read-only)
+NODE_NAME: TEST_DIR/t.IMGFMT (file)
+NODE_NAME: TEST_DIR/t.IMGFMT (file)
+NODE_NAME: TEST_DIR/t.IMGFMT (file)
+
+NODE_NAME: TEST_DIR/t.IMGFMT (file)
+NODE_NAME: TEST_DIR/t.IMGFMT (file)
+NODE_NAME: TEST_DIR/t.IMGFMT (file)

>> 235 and 238 apparently try to use accel=kvm, so they likely should not
>> be in the "quick" group.
> 
> Yes, I agree.
> 
> Or maybe we should check again why they need kvm at all, in theory they
> shouldn't. They are throttling tests, so the obvious reason seems to be
> that they need a running clock. Using -M qtest and advancing the clock
> manually could do the trick then (and would even be more reliable).
> 
>> The other tests seem to fail for different reasons, see:
>>
>>  https://gitlab.com/huth/qemu/-/jobs/163680780
>>
>> Some of them apparently need encryption to be enabled (as already
>> mentioned by Cleber in his patch) - thus should they really be in the
>> quick check, too? Or could they at least check whether QEMU has been
>> built with encryption?
> 
> The correct solution would be that they detect the situation
> automatically and skip the test by calling _notrun.
> 
> I'm not sure how to detect if a given QEMU binary supports encryption,
> but Dan might know.
> 
>> By the way, 235 and 238 also fail on my normal laptop with RHEL7:
>>
>> 235 2s ...        [07:29:31] [07:29:31] [failed, exit status 1] - output
>> mismatch (see 235.out.bad)
>> --- /home/thuth/devel/qemu/tests/qemu-iotests/235.out        2019-02-19
>> 07:14:45.000000000 +0100
>> +++ /home/thuth/tmp/qemu-build/tests/qemu-iotests/235.out.bad        
>> 2019-02-19
>> 07:29:31.000000000 +0100
>> @@ -1,3 +1,14 @@
>> -{"return": {}}
>> -{"return": {}}
>> -{"return": {}}
>> +Traceback (most recent call last):
>> +  File "235", line 56, in <module>
>> +    vm.launch()
>> +  File
>> "/home/thuth/devel/qemu/tests/qemu-iotests/../../scripts/qemu.py", line
>> 303, in launch
>> +    self._launch()
>> +  File
>> "/home/thuth/devel/qemu/tests/qemu-iotests/../../scripts/qemu.py", line
>> 329, in _launch
>> +    self._post_launch()
>> +  File
>> "/home/thuth/devel/qemu/tests/qemu-iotests/../../scripts/qemu.py", line
>> 274, in _post_launch
>> +    self._qmp.accept()
>> +  File
>> "/home/thuth/devel/qemu/tests/qemu-iotests/../../scripts/qmp/qmp.py",
>> line 157, in accept
>> +    return self.__negotiate_capabilities()
>> +  File
>> "/home/thuth/devel/qemu/tests/qemu-iotests/../../scripts/qmp/qmp.py",
>> line 73, in __negotiate_capabilities
>> +    raise QMPConnectError
>> +qmp.qmp.QMPConnectError
>> 236 0s ...        [07:29:31] [07:29:31]
>> 237                [07:29:31] [07:29:31] [not run]
>>         237 -- not suitable for this image format: qcow2
>> 238                [07:29:31] [07:29:31] [failed, exit status 1] -
>> output mismatch (see 238.out.bad)
>> --- /home/thuth/devel/qemu/tests/qemu-iotests/238.out        2019-02-19
>> 07:17:39.000000000 +0100
>> +++ /home/thuth/tmp/qemu-build/tests/qemu-iotests/238.out.bad        
>> 2019-02-19
>> 07:29:31.000000000 +0100
>> @@ -1,6 +1,14 @@
>> -{"return": {}}
>> -{"return": {}}
>> -{"return": {}}
>> -{"return": {}}
>> -{"return": {}}
>> -{"return": {}}
>> +Traceback (most recent call last):
>> +  File "238", line 37, in <module>
>> +    vm.launch()
>> +  File
>> "/home/thuth/devel/qemu/tests/qemu-iotests/../../scripts/qemu.py", line
>> 303, in launch
>> +    self._launch()
>> +  File
>> "/home/thuth/devel/qemu/tests/qemu-iotests/../../scripts/qemu.py", line
>> 329, in _launch
>> +    self._post_launch()
>> +  File
>> "/home/thuth/devel/qemu/tests/qemu-iotests/../../scripts/qemu.py", line
>> 274, in _post_launch
>> +    self._qmp.accept()
>> +  File
>> "/home/thuth/devel/qemu/tests/qemu-iotests/../../scripts/qmp/qmp.py",
>> line 157, in accept
>> +    return self.__negotiate_capabilities()
>> +  File
>> "/home/thuth/devel/qemu/tests/qemu-iotests/../../scripts/qmp/qmp.py",
>> line 73, in __negotiate_capabilities
>> +    raise QMPConnectError
>> +qmp.qmp.QMPConnectError
>>
>> Any ideas what might be going on here?
> 
> I think it's most likely that QEMU just prints an error message on
> startup and exits.
Right, I finally found the issue:

qemu-system-x86_64: -machine accel=kvm: No accelerator found

I apparently compiled my QEMU with --disable-kvm at one point in time and
forgot to enable it later again. ==> These tests should really check whether
KVM is available in QEMU before they blindly use this feature.

 Thomas