[Qemu-devel] [PATCH v6 00/12] block: drive-backup live backup command

[Stefan Hajnoczi]

Note: These patches apply to kevin/block.  You can also grab the code from git
here:
git://github.com/stefanha/qemu.git block-backup-core

This series adds a new QMP command, drive-backup, which takes a point-in-time
snapshot of a block device.  The snapshot is copied out to a target block
device.  A simple example is:

  drive-backup device=virtio0 format=qcow2 target=backup-20130401.qcow2

The original drive-backup blockjob was written by Dietmar Maurer
<address@hidden>.  He is currently busy but I feel the feature is worth
pushing into QEMU since there has been interest.  This is my version of his
patch, plus the QMP command and qemu-iotests test case.

QMP 'transaction' support is included since v3.  It adds support for atomic
snapshots of multiple block devices.  I also added an 'abort' transaction to
allow testing of the .abort()/.cleanup() code path.  Thanks to Wenchao for
making qmp_transaction() extensible.

How is this different from block-stream and drive-mirror?
---------------------------------------------------------
Both block-stream and drive-mirror do not provide immediate point-in-time
snapshots.  Instead they copy data into a new file and then switch to it.  In
other words, the point at which the "snapshot" is taken cannot be controlled
directly.

drive-backup intercepts guest writes and saves data into the target block
device before it is overwritten.  The target block device can be a raw image
file, backing files are not used to implement this feature.

How can drive-backup be used?
-----------------------------
The simplest use-case is to copy a point-in-time snapshot to a local file.

More advanced users may wish to make the target an NBD URL.  The NBD server
listening on the other side can process the backup writes any way it wishes.  I
previously posted an RFC series with a backup server that streamed Dietmar's
VMA backup archive format.

What's next for drive-backup?
-----------------------------
1. Sync modes like drive-mirror (top, full, none).  This makes it possible to
   preserve the backing file chain.

v6:
  * Note: I will send HMP support as a follow-up patch
  * Note: The 'Abort' action's name is unchanged
  * Extract wait_until_completed() into iotests.py
  * Populate image with non-zero data and resume job in 055 test_pause() [kwolf]
  * Document NotifierWithReturn return value semantics [eblake]
  * Convert DPRINTF() to trace events [pbonzini]
  * Avoid calling bdrv_getlength() inside the loop [kwolf]
  * Comment that BlockJob->offset is an opaque progress value [kwolf]
  * Align '=' to same column [kwolf]
  * Use bdrv_set_enable_write_cache(target, true) [kwolf]
  * Use BACKUP_SECTORS_PER_CLUSTERS instead of 1 sector for clarity [kwolf]
  * Check cancel_and_wait() 'data/type' is 'backup' in 055 [kwolf]

v5:
 * Use bdrv_co_write_zeroes(job->target) [kwolf]

   This change means that we write zeroes over NBD again.  The optimization can
   be reintroduced by skipping zeroes when bdrv_has_zero_init() is true and the
   sectors are allocated.  Leave that for a future series, if we decide to do
   this optimization again because it may require extra block driver
   configuration to indicate that an image has zero init.

 * iostatus error handling [kwolf/pbonzini]

   Add configurable on-source-error and on-target-error actions just like
   drive-mirror.  These are used when the block job coroutine hits an error.
   If we are in guest write request context, return the errno and let the usual
   guest error handling take over.

 * Allow BdrvActionOps->commit() to be NULL [eblake]
 * Use bdrv_getlength() in qmp_drive_mirror() [kwolf]
 * Drop redundant proto_drv check [kwolf]
 * Fix outdated DPRINTF() function names
 * Drop comment about non-existent bitmap coroutine race [kwolf]
 * Rename BACKUP_SECTORS_PER_CLUSTER [kwolf]
 * Fix completion when image is not multiple of backup cluster size [fam]

v4:
 * Use notifier lists and BdrvTrackedRequest instead of custom callbacks 
[bonzini]
 * Add drive-backup QMP example JSON [eblake]
 * Add "Since: 1.6" to QMP schema changes [eblake]

v3:
 * Rename to drive-backup for consistency with drive-mirror [kwolf]
 * Add QMP transaction support [kwolf]
 * Introduce bdrv_add_before_write_cb() to hook writes
 * Mention 'query-block-jobs' lists job of type 'backup' [eblake]
 * Rename rwlock to flush_rwlock [kwolf]
 * Fix space in block/backup.c comment [kwolf]

v2:
 * s/block_backup/block-backup/ in commit message [eblake]
 * Avoid funny spacing in QMP docs [eblake]
 * Document query-block-jobs and block-job-cancel usage [eblake]

Dietmar Maurer (1):
  block: add basic backup support to block driver

Stefan Hajnoczi (11):
  notify: add NotiferWithReturn so notifier list can abort
  block: add bdrv_add_before_write_notifier()
  blockdev: drop redundant proto_drv check
  blockdev: use bdrv_getlength() in qmp_drive_mirror()
  block: add drive-backup QMP command
  blockdev: rename BlkTransactionStates to singular
  blockdev: allow BdrvActionOps->commit() to be NULL
  blockdev: add DriveBackup transaction
  blockdev: add Abort transaction
  qemu-iotests: extract wait_until_completed() into iotests.py
  qemu-iotests: add 055 drive-backup test case

 block.c                       |  23 +--
 block/Makefile.objs           |   1 +
 block/backup.c                | 341 ++++++++++++++++++++++++++++++++++++++++++
 blockdev.c                    | 288 ++++++++++++++++++++++++++---------
 include/block/block_int.h     |  42 +++++-
 include/qemu/notify.h         |  29 ++++
 qapi-schema.json              |  97 +++++++++++-
 qmp-commands.hx               |  46 ++++++
 tests/qemu-iotests/041        |  14 +-
 tests/qemu-iotests/055        | 282 ++++++++++++++++++++++++++++++++++
 tests/qemu-iotests/055.out    |   5 +
 tests/qemu-iotests/group      |   1 +
 tests/qemu-iotests/iotests.py |  15 ++
 trace-events                  |   8 +
 util/notify.c                 |  30 ++++
 15 files changed, 1129 insertions(+), 93 deletions(-)
 create mode 100644 block/backup.c
 create mode 100755 tests/qemu-iotests/055
 create mode 100644 tests/qemu-iotests/055.out

-- 
1.8.1.4