Re: Fwd: [PATCH 0/2] block/raw: implemented persistent dirty bitmap and ability to dump bitmap content via qapi

[Patrik Janoušek]

On 3/22/21 1:06 PM, Max Reitz wrote:
> On 22.03.21 12:27, Patrik Janoušek wrote:
>> On 3/22/21 11:48 AM, Max Reitz wrote:
>>> Hi,
>>>
>>> On 20.03.21 11:01, Patrik Janoušek wrote:
>>>> I'm sorry, but I forgot to add you to the cc, so I'm forwarding the
>>>> patch to you additionally. I don't want to spam the mailing list
>>>> unnecessarily.
>>>
>>> I think it’s better to still CC the list.  It’s so full of mail, one
>>> more won’t hurt. :)
>>>
>>> (Re-adding qemu-block and qemu-devel, because the discussion belongs
>>> on the list(s).)
>>>
>>>> -------- Forwarded Message --------
>>>> Subject:     [PATCH 0/2] block/raw: implemented persistent dirty
>>>> bitmap and ability to dump bitmap content via qapi
>>>> Date:     Sat, 20 Mar 2021 10:32:33 +0100
>>>> From:     Patrik Janoušek <pj@patrikjanousek.cz>
>>>> To:     qemu-devel@nongnu.org
>>>> CC:     Patrik Janoušek <pj@patrikjanousek.cz>, lmatejka@kiv.zcu.cz
>>>>
>>>>
>>>>
>>>> Currently, QEMU doesn't support persistent dirty bitmaps for raw
>>>> format
>>>> and also dirty bitmaps are for internal use only, and cannot be
>>>> accessed
>>>> using third-party applications. These facts are very limiting
>>>> in case someone would like to develop their own backup tool becaouse
>>>> without access to the dirty bitmap it would be possible to implement
>>>> only full backups. And without persistent dirty bitmaps, it wouldn't
>>>> be possible to keep track of changed data after QEMU is restarted. And
>>>> this is exactly what I do as a part of my bachelor thesis. I've
>>>> developed a tool that is able to create incremental backups of drives
>>>> in raw format that are LVM volumes (ability to create snapshot is
>>>> required).
>>>
>>> Similarly to what Vladimir has said already, the thing is that
>>> conceptually I can see no difference between having a raw image with
>>> the bitmaps stored in some other file, i.e.:
>>>
>>>    { "driver": "raw",
>>>      "dirty-bitmaps": [ {
>>>        "filename": "sdc1.bitmap",
>>>        "persistent": true
>>>      } ],
>>>      "file": {
>>>        "driver": "file",
>>>        "filename": "/dev/sdc1"
>>>      } }
>>>
>>> And having a qcow2 image with the raw data stored in some other file,
>>> i.e.:
>>>
>>>    { "driver": "qcow2",
>>>      "file": {
>>>        "driver": "file",
>>>        "filename": "sdc1.metadata"
>>>      },
>>>      "data-file": {
>>>        "driver": "file",
>>>        "filename": "/dev/sdc1"
>>>      } }
>>>
>>> (Where sdc1.metadata is a qcow2 file created with
>>> “data-file=/dev/sdc1,data-file-raw=on”.)
>>>
>>> To use persistent bitmaps with raw images, you need to add metadata
>>> (namely, the bitmaps).  Why not store that metadata in a qcow2 file?
>>>
>>> Max
>>
>> So if I understand it correctly. I can configure dirty bitmaps in the
>> latest version of QEMU to be persistently stored in some other file.
>> Because even Proxmox Backup Server can't perform an incremental backup
>> after restarting QEMU, and that means something to me. I think they
>> would implement it if it was that simple.
>>
>> Could you please send me simple example on how to configure (via command
>> line args) one raw format drive that can store dirty bitmaps
>> persistently in other qcow2 file? I may be missing something, but I
>> thought QEMU couldn't do it, because Proxmox community wants this
>> feature for a long time.
>
> One trouble is that if you use qemu-img create to create the qcow2
> image, it will always create an empty image, and so if use pass
> data_file to it, it will empty the existing raw image:
>
> $ cp ~/tmp/arch.iso raw.img # Just some Arch Linux ISO
>
> $ qemu-img create \
>     -f qcow2 \
>     -o data_file=raw.img,data_file_raw=on,preallocation=metadata \
>     metadata.qcow2 \
>     $(stat -c '%s' raw.img)
> Formatting 'metadata.qcow2', fmt=qcow2 cluster_size=65536
> preallocation=metadata compression_type=zlib size=687865856
> data_file=raw.img data_file_raw=on lazy_refcounts=off refcount_bits=16
>
> (If you check raw.img at this point, you’ll find that it’s empty, so
> you need to copy it from the source again:)
>
> $ cp ~/tmp/arch.iso raw.img
>
> Now if you use metadata.qcow2, the image data will actually all be
> stored in raw.img.
>
>
> To get around the “creating metadata.qcow2 clears raw.img” problem,
> you can either create a temporary empty image of the same size as
> raw.img that you pass to qemu-img create, and then you use qemu-img
> amend to change the data-file pointer (which will not overwrite the
> new data-file’s contents):
>
> $ qemu-img create -f raw tmp.raw $(stat -c '%s' raw.img)
>
> $ qemu-img create \
>     -f qcow2 \
>     -o data_file=tmp.img,data_file_raw=on,preallocation=metadata \
>     metadata.qcow2 \
>     $(stat -c '%s' raw.img)
> Formatting 'metadata.qcow2', fmt=qcow2 cluster_size=65536
> preallocation=metadata compression_type=zlib size=687865856
> data_file=tmp.img data_file_raw=on lazy_refcounts=off refcount_bits=16
>
> $ qemu-img amend -o data_file=raw.img metadata.qcow2
>
> $ rm tmp.img
>
>
> Or you use the blockdev-create job to create the qcow2 image (because
> contrary to qemu-img create, that will not clear the data file):
>
> $ touch metadata.qcow2
>
> (Note that in the following QMP communication, what I sent and what
> qemu replies is mixed.  Everything that begins with '{ "execute"' is
> from me, everything else from qemu.  The number 687865856 is the size
> of raw.img in bytes.)
>
> $ qemu-system-x86_64 -qmp stdio \
>     -blockdev \
>     '{ "node-name": "metadata-file",
>        "driver": "file",
>        "filename": "metadata.qcow2" }' \
>     -blockdev \
>     '{ "node-name": "data-file",
>        "driver": "file",
>        "filename": "raw.img" }'
> {"QMP": {"version": {"qemu": {"micro": 0, "minor": 1, "major": 5},
> "package": "qemu-5.1.0-9.fc33"}, "capabilities": ["oob"]}}
>
> { "execute": "qmp_capabilities" }
> {"return": {}}
>
> { "execute": "blockdev-create",
>   "arguments": {
>     "job-id": "create",
>     "options": {
>       "driver": "qcow2",
>       "file": "metadata-file",
>       "data-file": "data-file",
>       "data-file-raw": true,
>       "preallocation": "metadata",
>       "size": 687865856
>     }
>   }
> }
> {"timestamp": {"seconds": 1616414002, "microseconds": 836899},
> "event": "JOB_STATUS_CHANGE", "data": {"status": "created", "id":
> "create"}}
> {"timestamp": {"seconds": 1616414002, "microseconds": 837076},
> "event": "JOB_STATUS_CHANGE", "data": {"status": "running", "id":
> "create"}}
> {"return": {}}
> {"timestamp": {"seconds": 1616414002, "microseconds": 870997},
> "event": "JOB_STATUS_CHANGE", "data": {"status": "waiting", "id":
> "create"}}
> {"timestamp": {"seconds": 1616414002, "microseconds": 871099},
> "event": "JOB_STATUS_CHANGE", "data": {"status": "pending", "id":
> "create"}}
> {"timestamp": {"seconds": 1616414002, "microseconds": 871185},
> "event": "JOB_STATUS_CHANGE", "data": {"status": "concluded", "id":
> "create"}}
>
> { "execute": "job-dismiss", "arguments": { "id": "create" } }
> {"timestamp": {"seconds": 1616414022, "microseconds": 202880},
> "event": "JOB_STATUS_CHANGE", "data": {"status": "null", "id": "create"}}
> {"return": {}}
>
> { "execute": "quit" }
> {"return": {}}
> {"timestamp": {"seconds": 1616414028, "microseconds": 56457}, "event":
> "SHUTDOWN", "data": {"guest": false, "reason": "host-qmp-quit"}}
>
>
> In any case, in the end you get a metadata.qcow2 that holds the
> metadata and points to raw.img for its data.  So you can use it like
> this:
>
> $ qemu-system-x86_64 -enable-kvm -m 512 \
>     -blockdev \
>     '{ "node-name": "node0",
>        "driver": "qcow2",
>        "file": {
>          "driver": "file",
>          "filename": "metadata.qcow2"
>        } }' \
>     -device ide-cd,drive=node0 \
>     -qmp stdio
> {"QMP": {"version": {"qemu": {"micro": 0, "minor": 1, "major": 5},
> "package": "qemu-5.1.0-9.fc33"}, "capabilities": ["oob"]}}
>
> { "execute": "qmp_capabilities" }
> {"return": {}}
>
> { "execute": "block-dirty-bitmap-add",
>   "arguments": {
>     "node": "node0",
>     "name": "bmap0",
>     "persistent": true
>   }
> }
> {"return": {}}
>
> { "execute": "quit" }
> {"return": {}}
> {"timestamp": {"seconds": 1616414627, "microseconds": 928250},
> "event": "SHUTDOWN", "data": {"guest": false, "reason": "host-qmp-quit"}}
>
> $ qemu-img info metadata.qcow2
> image: metadata.qcow2
> file format: qcow2
> virtual size: 656 MiB (687865856 bytes)
> disk size: 452 KiB
> cluster_size: 65536
> Format specific information:
>     compat: 1.1
>     compression type: zlib
>     lazy refcounts: false
>     bitmaps:
>         [0]:
>             flags:
>                 [0]: auto
>             name: bmap0
>             granularity: 65536
>     refcount bits: 16
>     data file: raw.img
>     data file raw: true
>     corrupt: false
>
> So the bitmap is now in metadata.qcow2, and as the disk size
> indicates, all the data is still in raw.img.
>
>
> I hope the above helps you.  Sorry if it’s confusing, especially the
> first part where I’m like “The obvious way to create metadata.qcow2
> will delete your data, so here are two alternatives that are weird but
> do what you want.”
>
> Max

Thank you for exhausting answer. I would also like to thank others for
their comments. I will try to rethink the developed solution and
consider its modifications. Maybe in the future I'll come up with
something you might like.

Regards,
Patrik