Re: [Qemu-block] 4.2 CI failure on qemu-io tests

[Milan Zamazal]

Nir Soffer <address@hidden> writes:

> On Fri, Dec 7, 2018 at 3:19 PM Milan Zamazal <address@hidden> wrote:
>
>> Hi, I've seen a CI failure on qemu-io tests in ovirt-4.2 branch:
>> http://jenkins.ovirt.org/job/vdsm_standard-check-patch/602/
>
>
> We have several issues:

Thank you Nir for looking.

> *00:12:02.358*     def verify_pattern(path, format, offset=512,
> len=1024, pattern=5):*00:12:02.358*         read_cmd = 'read -P %d -s
> 0 -l %d %d %d' % (pattern, len, offset, len)*00:12:02.358*         cmd
> = ['qemu-io', '-f', format, '-c', read_cmd, path]*00:12:02.358*
>  rc, out, err = commands.execCmd(cmd, raw=True)*00:12:02.358*
> if rc != 0 or err != b"":*00:12:02.358* >           raise
> cmdutils.Error(cmd, rc, out, err)*00:12:02.358* E           Error:
> Command ['qemu-io', '-f', 'qcow2', '-c', 'read -P 240 -s 0 -l 1024 0
> 1024', 
> '/var/tmp/tmpbxNeWA/mnt/blockSD/3fd2ac6f-929b-425a-a7d7-3b8519dedc91/images/e438933b-7d87-4fd6-a2fa-e53cb2e26a19/8ff32e26-b426-4ebf-a8cb-48353cf898e7']
> failed with rc=1 out='Pattern verification failed at offset 0, 1024
> bytes\nread 1024/1024 bytes at offset 0\n1 KiB, 1 ops; 0.0000 sec
> (17.756 MiB/sec and 18181.8182 ops/sec)\n' err=''
>
>
> This is the same issue fixed in master by:
>
> commit a341dcc3ab5c36afc7a536d3d4794c37fedcda6d
> Author: Nir Soffer <address@hidden>
> Date:   Sun Nov 25 22:48:54 2018 +0200
>
>     qemuio: Support qemu 2.12 error handling

>     qemu-io 2.10 was exiting with zero exit code and "Pattern verification"
>     error in stdout. In 2.12, non-zero code is returned when pattern
>     verification fails. This sounds better, but the exit code is not
>     specific enough to detect the verification error.
>
>     Check for pattern verification error before testing exit code, so we
>     raise (expected) verification error in this case, and raise a generic
>     error in verification error was not detected.
>
>     Change-Id: Id108d2b780b6862bed3bb7088908d911788b544d
>     Signed-off-by: Nir Soffer <address@hidden>
>
> Backporting this patch should fix this issue.

Backported, fixed.

> *00:12:02.352* Error: Command ['/usr/bin/taskset', '--cpu-list',
> '0-1', '/usr/bin/nice', '-n', '19', '/usr/bin/ionice', '-c', '3',
> '/usr/bin/qemu-img', 'convert', '-p', '-t', 'none', '-T', 'none',
> '-f', 'raw', 
> '/var/tmp/tmpmwqNI9/9a7a895a-eaed-4de0-97b5-05009e336853/15ca6f4c-2753-450f-8e4b-1e3278fc664e/images/89432c41-c8c9-4d4e-ab86-ef892806162b/9dbca6ff-412a-4c85-a4eb-51dfa4fadbd8',
> '-O', 'raw', 
> '/var/tmp/tmpmwqNI9/9a7a895a-eaed-4de0-97b5-05009e336853/15ca6f4c-2753-450f-8e4b-1e3278fc664e/images/51e0d2ba-0cb0-421c-901b-4842e8612ca4/3b857f90-3062-4698-9920-2c4849b7256f']
> failed with rc=-6 out='' err=bytearray(b"qemu-img: block/io.c:2134:
> bdrv_co_block_status: Assertion `*pnum && (((*pnum) % (align)) == 0)
> && align > offset - aligned_offset\' failed.\n")
>
>
> We found this error 6 month ago on travis, and discussed this here:
> https://lists.ovirt.org/archives/list/address@hidden/thread/UGGNDS4SVJ2GHAQ57H2PZWMFC33ODLA4/#TF6BPZIIJYMFJLFFJYGADKBQ2FDH5Q7O
>
> We filed
> https://bugzilla.redhat.com/1589738
>
> Then we fixed the issue in the tests by using aligned images, since we
> always use
> aligned images in the real system.
>
> commit 0cde33b4ecfcc55dee3f441749fc72c106b4dd77
> Author: Nir Soffer <address@hidden>
> Date:   Sun Jun 24 22:14:57 2018 +0300
>
>     tests: Use aligned image size
>
>     The qcow2 tests used unaligned image size, which is not realistic test
>     data, and triggered a segfault in qemu-img on Fedora 28. While exposing
>     bugs in qemu-img is nice, this is not the place for such test.
>
>     Change-Id: I2dca62187dce72638c3f92a4694b8f287d7a048f
>     Signed-off-by: Nir Soffer <address@hidden>

Not necessarily needed, but I'd take it anyway, see my comment in
https://gerrit.ovirt.org/96224 .

> commit 3e43fff81f51002347277df83ecfff61d5c8a4a5
> Author: Nir Soffer <address@hidden>
> Date:   Sun Jun 24 22:01:21 2018 +0300
>
>     tests: Use padded VM configuration volume
>
>     The tests was using unpadded vm configuration volume. This causes
>     qemu-img to segfault in Fedora 28. Since we always pad vm configuration
>     volumes in production code, we don't need to test unpadded values.
>
>     Change-Id: I74cb26dcc0c8633fa1de5f689d2e8ffa98013d48
>     Signed-off-by: Nir Soffer <address@hidden>
>
> Backporting these patches should fix the failing tests.

The second one backported, CI fixed.

> The qemu bug was fixed few month after we reported it.
>
> The fix is available in:
> qemu-img.x86_64 2:2.11.2-1.fc28
> qemu-img.x86_64 2:3.0.0-1.fc28 (from virt-preview)
>
> Are we using these versions in 4.2 CI? Do we have correct repos?

I think so.  4.2 CI should be using 2.12.0-18.el7_6.1.1 from
http://mirror.centos.org/centos/7/virt/x86_64/kvm-common/.  And that it
fails without "qemuio: Support qemu 2.12 error handling" confirms that
2.12 is indeed used.

> Strangely I see that we have also this bug, which was probably
> tested on RHEL, but there is on info on the versions tested.
> https://bugzilla.redhat.com/1649788
>
> So it seems that this bug sneaked into some downstream versions.
>
> Nir