Allocation change in qemu-img convert (since 5.1?)

[Nir Soffer]

We have tests for vdsm qemuimg wrapper module converting
images and verifying the operation using "qemu-img compare".

I found that tests using "qemu-img compare -s" started to fail
recently on RHEL 8.3.0 nightly
(qemu-img-5.1.0-10.module+el8.3.0+8254+568ca30d.x86_6)
and Fedora 32 using virt-preview (qemu-img-5.1.0-5.fc32.x86_64)

Here is an example reproducing this issue on ext4 filesystem
on Fedora 32. I did check this case on RHEL 8.3.0, but I think the
same test fails there.

Create source chain:

$ qemu-img create -f qcow2 src_base.qcow2 10m
Formatting 'src_base.qcow2', fmt=qcow2 cluster_size=65536
compression_type=zlib size=10485760 lazy_refcounts=off
refcount_bits=16

$ qemu-img create -f qcow2 src_top.qcow2 -b src_base.qcow2 -F qcow2
Formatting 'src_top.qcow2', fmt=qcow2 cluster_size=65536
compression_type=zlib size=10485760 backing_file=src_base.qcow2
backing_fmt=qcow2 lazy_refcounts=off refcount_bits=16

Create destination chain:

$ qemu-img create -f qcow2 dst_base.qcow2 10m
Formatting 'dst_base.qcow2', fmt=qcow2 cluster_size=65536
compression_type=zlib size=10485760 lazy_refcounts=off
refcount_bits=16

$ qemu-img create -f qcow2 dst_top.qcow2 -b dst_base.qcow2 -F qcow2
Formatting 'dst_top.qcow2', fmt=qcow2 cluster_size=65536
compression_type=zlib size=10485760 backing_file=dst_base.qcow2
backing_fmt=qcow2 lazy_refcounts=off refcount_bits=16

Write data to source chain:

$ qemu-io -f qcow2 -c 'write -P 240 0 65536' src_base.qcow2
wrote 65536/65536 bytes at offset 0
64 KiB, 1 ops; 00.01 sec (7.355 MiB/sec and 117.6809 ops/sec)

$ qemu-io -f qcow2 -c 'write -P 241 1048576 65536' src_top.qcow2
wrote 65536/65536 bytes at offset 1048576
64 KiB, 1 ops; 00.01 sec (7.509 MiB/sec and 120.1387 ops/sec)

Compare images:

$ qemu-img compare src_base.qcow2 dst_base.qcow2
Images are identical.

$ qemu-img compare src_base.qcow2 dst_base.qcow2 -s
Strict mode: Offset 65536 block status mismatch!

This check succeeds with qemu-img 4.2.0-19 in our Travis CI.

$ qemu-img map --output json src_base.qcow2
[{ "start": 0, "length": 65536, "depth": 0, "zero": false, "data":
true, "offset": 327680},
{ "start": 65536, "length": 10420224, "depth": 0, "zero": true, "data": false}]

$ qemu-img map --output json dst_base.qcow2
[{ "start": 0, "length": 65536, "depth": 0, "zero": false, "data":
true, "offset": 327680},
{ "start": 65536, "length": 10420224, "depth": 0, "zero": true, "data": false}]

Allocation looks identical - the first extent is data, and the second is a hole.

But if we check using qemu-nbd[1]:

$ python
>>> from ovirt_imageio import client
>>> from pprint import pprint
>>> pprint(list(client.extents("src_base.qcow2")))
[ZeroExtent(start=0, length=65536, zero=False, hole=False),
 ZeroExtent(start=65536, length=10420224, zero=True, hole=True)]
>>> pprint(list(client.extents("dst_base.qcow2")))
[ZeroExtent(start=0, length=65536, zero=False, hole=False),
 ZeroExtent(start=65536, length=10420224, zero=True, hole=False)]

This shows an issue - the second extent is reported as a hole in the
source image, and as an allocated area in the destination image.

$ ls -lhs *_base.qcow2
324K -rw-r--r--. 1 nsoffer nsoffer 384K Oct 25 17:12 dst_base.qcow2
324K -rw-r--r--. 1 nsoffer nsoffer 384K Oct 25 17:10 src_base.qcow2

Maybe the clusters in the source image are zeroed? But if they are
zeroed, why qemu-img map reports them as a hole? Seems that we have
a behavior change in qemu-img convert, and a bug in qemu-img map.

I compared also the top images, after removing the backing file:

$ qemu-img rebase -u -b "" src_top.qcow2
$ qemu-img rebase -u -b "" dst_top.qcow2
$ qemu-img compare src_top.qcow2 dst_top.qcow2
Images are identical.
$ qemu-img compare src_top.qcow2 dst_top.qcow2 -s
Images are identical.

So the issue is only when converting single image to single image
without a backing file.

[1] export the image with qemu-nbd, and list the extents using
NBD_CMD_BLOCK_STATUS. Using ovirt-imageio-client 2.1.1.

Nir