[Qemu-devel] [PATCH 0/9] QMP: Introduce incremental drive-backup with in-memory dirty bitmap

[Fam Zheng]

This implements incremental backup.

A few new QMP commands related to dirty bitmap are added:

    dirty-bitmap-add *
    dirty-bitmap-disable *
    dirty-bitmap-remove

    (*: also supported as transactions)

As their name implies, they manipulate a block device's dirty bitmap. This
doesn't interfere with dirty bitmap used for migration, backup, mirror, etc,
which don't have a name and are invisible to user. Only named bitmaps (created
by dirty-bitmap-add) can be disabled/removed by user.

They are added to support "user controlled write tracking", so as to determine
the range of date for incremental backup.

A new sync mode for drive-backup is introduced:

    drive-backup device=.. mode=.. sync=dirty-bitmap bitmap=bitmap0

Which will scan dirty bitmap "bitmap0" and only copy all dirty sectors to
target.

Now, let's see the usage with a simple example:

    # Start the guest
    vm = VM()
    vm.start()

    # Fake some guest writes with "qemu-io", this is before creating dirty
    # bitmap, so it won't be copied
    vm.hmp('qemu-io ide0-hd0 "write -P 0xa 512k 1M"')

    # Create a dirty bitmap to track writes
    vm.qmp("dirty-bitmap-add", device="ide0-hd0", name="dirty-0")

    # Fake some more guest writes with "qemu-io", this will be copied
    vm.hmp('qemu-io ide0-hd0 "write -P 0xa 512M 1M"')

    # Now "disable" the first dirty bitmap, do the backup according to it,
    # at meantime continue to track dirty with a new dirty bitmap
    vm.qmp("transaction", actions=[
        {
            'type': 'dirty-bitmap-disable', 'data': {
                'device': 'ide0-hd0',
                'name': 'dirty-0'
            }
        }, {
            'type': 'dirty-bitmap-add', 'data': {
                'device': 'ide0-hd0',
                'name': 'dirty-1'
                }
        }, {
            'type': 'drive-backup', 'data': {
                'device': 'ide0-hd0',
                'target': '/tmp/incre.qcow2',
                'bitmap': 'dirty-0',
                'sync': 'dirty-bitmap'
                }
            }
        ])

    # Once backup job started, the dirty bitmap can be removed (actually only
    # hidden from user since it is still referenced by block job
    vm.qmp("dirty-bitmap-remove", device="ide0-hd0", name="dirty-0")

Wait the block job to complete, then let's check the target image to see what
data is copied:

    ./qemu-img map /tmp/incre.qcow2
    Offset          Length          Mapped to       File
    0x20000000      0x100000        0x160000        /tmp/incre.qcow2

Yes, only the data written after "dirty0" creation is copied.

If this interface looks good, test cases will be included in the next revision.

Also, it's quite easy to use an rbd server or other protocols to do the remote
backup over network.

P.S. Persistent dirty bitmap could be built on top of this series, but is not
yet implemented, because the storage format and integrity measures are not
quite clear for now. The discussion is still open and any questions, ideas, use
cases and concerns are all welcome!

Fam


Fam Zheng (9):
  block: Introduce reference count for dirty bitmaps
  qapi: Add optional field "name" to block dirty bitmap
  block: Add bdrv_dirty_bitmap_make_anon
  qmp: Add dirty-bitmap-add and dirty-bitmap-remove
  block: Handle error of bdrv_getlength in bdrv_create_dirty_bitmap
  block: Introduce bdrv_dirty_bitmap_granularity()
  block: Add support of "dirty-bitmap" sync mode
  qmp: Add dirty-bitmap-disable command
  qapi: Add transaction support to dirty-bitmap-{add,disable}

 block-migration.c         |   3 +-
 block.c                   |  79 +++++++++++++++++++++--
 block/backup.c            |  34 +++++++++-
 block/mirror.c            |   6 +-
 blockdev.c                | 156 +++++++++++++++++++++++++++++++++++++++++++++-
 hmp.c                     |   3 +-
 include/block/block.h     |  13 +++-
 include/block/block_int.h |   2 +
 qapi-schema.json          |  79 +++++++++++++++++++++--
 qmp-commands.hx           |  61 +++++++++++++++++-
 10 files changed, 416 insertions(+), 20 deletions(-)

-- 
1.8.5.2