Re: [Qemu-devel] While testing `drive-backup` with 'dirty-bitmap' syc mode via QMP. . .

[Kashyap Chamarthy]

On Tue, Apr 21, 2015 at 11:34:07AM -0400, John Snow wrote:
> Kashyap Chamarthy sent me a question asking for some clarifications on
> testing my incremental backup series. I responded off-list but he has asked
> me to reproduce my answer for the archives :)

Thanks for the detailed response on both online/offline cases further
below.

Just as a side note, I wrote a short version of the same email to the
list to your v 2.5 thread. I think you sent this response before
noticing them.  Sorry for the double emails. :-)

    http://lists.nongnu.org/archive/html/qemu-devel/2015-04/msg02384.html
    http://lists.nongnu.org/archive/html/qemu-devel/2015-04/msg02406.html

So for the record, your details below should give clues to answer those.

[. . .]

> >Test
> >----
> >
> >(1) I have the QEMU invocation w/ QMP of a CirrOS disk image, that way:
> >
> >   
> > /home/kashyapc/tinker-space/qemu-upstream/x86_64-softmmu/qemu-system-x86_64 
> > \
> >   . . .
> >   -drive 
> > file=/export/full_backup.img,if=none,id=drive-ide0-0-0,format=qcow2 \
> >   -qmp tcp:localhost:4444,server
> >
> >So, it's waiting on the server:
> >
> >   $ ./2-min-qemu-invoc-qmp.sh
> >   char device redirected to /dev/pts/31 (label charserial0)
> >   QEMU waiting for connection on: disconnected:tcp:localhost:4444,server
> >
> >
> >(2) Then, when I execute the `drive-backup` QMP command:
> >-----------------------------------
> >#!/bin/bash
> >set -x
> ># Test incremental backup with `drive-backup`
> >
> >exec 3<>/dev/tcp/localhost/4444
> >echo -e "{ 'execute': 'qmp_capabilities' }" >&3
> >read response <&3
> >echo $response
> >echo -e "{ 'execute': 'drive-backup', 'arguments':
> >       { 'device': 'drive-ide0-0-0', 'bitmap': 'bitmap0', 'sync': 
> > 'dirty-bitmap', 'target':
> >       
> > '/home/kashyapc/work/virt/qemu/incremental-backup-test-qemu/tests/incremental.0.img',
> >  'mode': 'existing', 'format': 'qcow2' } }" >&3
> >read response <&3
> >echo $response
> >-----------------------------------
> >
> >(3) Invoke it:
> >
> >$ ./invoke-drive-backup.sh
> >{"QMP": {"version": {"qemu": {"micro": 93, "minor": 2, "major": 2}, 
> >"package": ""}, "capabilities": []}}
> >{"return": {}}
> >
> >
> >Now, I don't see any feedback notification on the QMP server as a result
> >of the `drive-bacup` execution from step (3) unlike from a slightly old
> >test of regular `drive-backup` command[1].
> >
> >NOTE: I tested with and without pre-creating the 'incremental.0.img'
> >file, to no avail.
> >
> >Am I missing anything obvious?
> >
> >
> >[1] https://kashyapc.fedorapeople.org/virt/test-qmp-drive-backup.txt
> >
> 
> (1) It doesn't appear as though you've created a bitmap in this example?

You're right! Shame on me for missing the important bit. I should have:

    { 'execute': 'block-dirty-bitmap-add', 'arguments': { 'node': 
'drive-ide0-0-0', 'name': 'bitmap0' } }


> (2) Incremental backup is designed to work basically exclusively with
> 'mode': 'existing' because it relies on the user (or the management
> utility/api) to keep track of what the "last backup" was.
> 
> (I had actually attempted to add that convenience feature into QEMU, but the
> sense was that the complexities of that were best left to libvirt, and I
> agreed.)
> 
> Here's the workflow (if the VM has not begun guest execution)
>   (A) Create a bitmap ("bitmap0" is my usual choice)
>   (B) Create an "anchor" or "tail" backup:
>       This is a full backup of the drive, performed as usual.
>   (C) Create the image to be used, using e.g. tail.qcow2 as
>       the backing file.
>   (D) Ensure that <bitmap0> is clear by issuing a clear command. Look
>       at bitmaps.md for an example.
>   (E) You may optionally start the VM to allow the disk to change
>       at this point.
>   (F) You may now issue an incremental backup command:
>       { 'execute': 'drive-backup',
>         'arguments': {
>           'device': 'drive-ide0-0-0',
>           'bitmap': 'bitmap0',
>           'sync': 'dirty-bitmap',
>           'target': .../tail.qcow2,
>           'mode': 'existing',
>           'format': 'qcow2'
>         }
>       }
> 
> The workflow is different if the VM has already started,
> for data safety reasons. (We need extra precautions to ensure
> the coherency of the incremental backup.)
> 
>   (A) Create a bitmap. ("bitmap0")
>   (B) Using a single transaction, you must accomplish the following:
>     (i) Create a full backup of 'drive-ide0-0-0'
>     (ii) Clear bitmap0.
>          This ensures that the bitmap tracks only writes that have
>          occurred after the full backup was made. This is why the
>          transactions are important.
> 
>          It's OK to let QEMU create the tail.qcow2 file here,
>          so we don't need mode:existing.
> 
>          { 'execute': 'transaction',
>            'arguments': {
>              'actions': [
>                { 'type': 'block-dirty-bitmap-clear',
>                  'data': {
>                    "node": 'drive-ide0-0-0',
>                    "name": 'bitmap0'
>                  }
>                },
>                { 'type': 'drive-backup',
>                  'data': {
>                    'device': 'drive-ide0-0-0',
>                    'bitmap': 'bitmap0',
>                    'sync': 'full',
>                    'target': .../tail.qcow2,
>                    'format': 'qcow2'
>                  }
>                }
>              ]
>            }
>          }
> 
>   (C) We need to create the incremental backup image outside of QEMU;
>       # qemu-img create -f qcow2 -F qcow2 -b .../tail.qcow2
>   (D) Now that we have a full backup and a synced bitmap, we can
>       issue a standard incremental backup command at any time.
>       { 'execute': 'drive-backup',
>         'arguments': {
>           'device': 'drive-ide0-0-0',
>           'bitmap': 'bitmap0',
>           'sync': 'dirty-bitmap',
>           'target': .../tail.qcow2,
>           'mode': 'existing',
>           'format': 'qcow2'
>         }
>       }
> 
> (3) You are probably not seeing any return from this script because it looks
> as if you only read from the QMP stream once, which will just give you the
> reply from accepting the QMP command, but the event that tells you if it
> completed successfully or not occurs later, so you need one more read. If
> you use bash to send/receive QMP commands, you need to poll the fd to get
> event responses that occur later.

Thanks for these details! And, also didn't know of the "poll the fd"
bit you mention above below. As this shows, I just began poking at this,
so I'll obviously have to test more with different cases.


-- 
/kashyap