[Qemu-block] [PATCH v6 00/21] block: transactionless incremental backup series

[John Snow]

It's spring! The winter snow has thawed and so here is a new
patch series to enter your life and warm your heart.

This patchset enables the in-memory part of the incremental backup
feature, without transactional support.

Support for transactions was separated into a separate series which
is also now available on-list. Getting this portion of the series
committed will help stabilize work on bitmap persistence and bitmap
migration.

Thanks to Fam Zheng for the original versions of this patchset,
And thanks to Max and Eric for reviewing 2,396 versions of it since.

===
v6:
===

01: s/underlaying/underlying/
    Removed a reference to 'disabled' bitmaps.
    Touching up inconsistent list indentation.
    Added FreeBSD Documentation License, primarily to be difficult
07: More in-line documentation for hbitmap_merge, for return value.
    Fix size cache index to be uint64_t.
09: Grammar fixes from Eric Blake, kept R-Bs.
10: Moved yield into the do{}while(). Now we check to see if we should
       yield/cancel after each unsuccessful sector we transfer.
    Some documentation additions for Eric Blake.
15: corrected 'num_elements' to 'start'
18: Refactored qmp.py event functions,
    Added in more explicit exception classes.
    No changes to iotests.py, just qmp.py.

Key:
[----] : patches are identical
[####] : number of functional differences between upstream/downstream patch
[down] : patch is downstream-only
The flags [FC] indicate (F)unctional and (C)ontextual differences, respectively

001/21:[0057] [FC] 'docs: incremental backup documentation'
002/21:[----] [--] 'qapi: Add optional field "name" to block dirty bitmap'
003/21:[----] [--] 'qmp: Ensure consistent granularity type'
004/21:[----] [--] 'qmp: Add block-dirty-bitmap-add and 
block-dirty-bitmap-remove'
005/21:[----] [--] 'block: Introduce bdrv_dirty_bitmap_granularity()'
006/21:[----] [--] 'hbitmap: cache array lengths'
007/21:[0008] [FC] 'hbitmap: add hbitmap_merge'
008/21:[----] [--] 'block: Add bitmap disabled status'
009/21:[0008] [FC] 'block: Add bitmap successors'
010/21:[0013] [FC] 'qmp: Add support of "dirty-bitmap" sync mode for 
drive-backup'
011/21:[----] [--] 'qmp: add block-dirty-bitmap-clear'
012/21:[----] [--] 'qmp: Add dirty bitmap status field in query-block'
013/21:[----] [--] 'block: add BdrvDirtyBitmap documentation'
014/21:[----] [--] 'block: Ensure consistent bitmap function prototypes'
015/21:[0002] [FC] 'block: Resize bitmaps on bdrv_truncate'
016/21:[----] [--] 'hbitmap: truncate tests'
017/21:[----] [-C] 'iotests: add invalid input incremental backup tests'
018/21:[0095] [FC] 'iotests: add QMP event waiting queue'
019/21:[----] [--] 'iotests: add simple incremental backup case'
020/21:[----] [--] 'iotests: add incremental backup failure recovery test'
021/21:[----] [--] 'iotests: add incremental backup granularity tests'

===
v5:
===

10: Code has been moved into backup_run_incremental()
    'polyrhythm' check is removed,
        clusters_per_iter variable is introduced instead.
    If the bitmap granularity is larger than the backup granularity,
        loop over the backup_do_cow call multiple times.
    If the bitmap granularity is smaller, skip the iterator ahead as
        we had been doing previously.
14: Only whitespace changes caused by patch 10.
15: Changed my approach for clearing out data for the hbitmap
        truncate shrink case, as suggested by Stefan
18: Added a proper timeout mechanism to qmp.pull_event():
      wait=False or wait=0.0 implies non-blocking.
      wait=True implies blocking.
      wait=60.0 implies a 60 second timeout.
      VM.event_wait() now uses a 60 second timeout by default.
19: Many things:
    The big picture is to add a set of full backups alongside the
    incremental backups created during the test to be able to test
    the validity of each incremental at the conclusion of the test
    when we can shut the VM down.

    To do this, there are two basic changes:
    (1) Keep a list of pairs of backup filenames (incremental, reference);
        create a full reference backup for every incremental created.
    (2) Refactor the backup helper functions a bit.
20: Naming fallout from 19
    Added calls to vm.shutdown() and check_backups().
21: NEW, adds granularity tests that cover the changes in patch 10.

===
v4:
===

04: Some in-line documentation for block_dirty_bitmap_lookup
    Changed behavior with respect to aio_context
      (always acquire, release if pbs == null, give to user otherwise)
10: Removed vestigial (currently nop) bdrv_enable_dirty_bitmap call
    Kept R-B.
16: Added some comments to test_check_boundary_bits.
    Kept R-B.
17: Folded in refactor from "incremental transactions v1" (Poor Kitty)
18: Pulled forward from "incremental transactions v1"
    Kept R-B from that series.
19: Folded in refactor from "incremental transactions v1"
    Added offset assertions into wait_incremental
20: Removed error tolerance from wait_until_completed, as
    these patches no longer make use of that feature.

===
v3:
===

01: Removed enabled/disabled modes information.
    Elaborated on events that can occur during error cases.
04: Added an AioContext out parameter to block_dirty_bitmap_lookup.
06: NEW:
    Cache the array lengths for hbitmap.
07: hbitmap_merge now uses the cached array lengths.
11: block-dirty-bitmap-clear is edited for the new block_dirty_bitmap_lookup.
12: Removed the "disabled" status, leaving just "Frozen."
15: Moved bdrv_truncate_dirty_bitmap to be static local
    Inlined dirty_bitmap_truncate function.
    Removed size[] caching into new patch (06, above)
    hbitmap_truncate now keeps correct bit population count
    hbitmap_truncate now uses hbitmap_reset BEFORE the truncate,
        to avoid tricky out-of-bounds usages.
    Remove g_realloc_n call that is not available in glib 2.12 (or 2.22)
    Renamed "truncate" to "shrink" to make that more clear
        to people who aren't me (at last count: 7+ billion)
16 NEW:
   hbitmap_truncate tests.

===
v2:
===

01: Added a new opening blurb.
    Adjusted codeblock indentations to be 4 spaces instead of 3,
    so it works as MD or GFMD.
    Adjusted errors explanation.
    Make visual separations between json data and shell commands
    Eliminate any ligering single quotes

07: Remember that memset takes bytes, not n_items ...

===
v1:
===

Deletions:
 - Removed Transactions, to be added later.
 - Removed Transaction tests, as above.

Changes:
01: Indentation fixes.
    Removed enable/disable documentation.
    Added a note that transactions aren't implemented yet.
    Removed my needless commas
    Added error case documentation.

07: QMP enable/disable commands are deleted.

14: Some comments concerning assertions.
    Scrub re-alloc memory if we expand the array.
    Do not attempt to scrub memory if fix_count is 0

Changes made with Reviews kept:

02: Since 2.4
04: Since 2.4
    Demingled the QMP command documentation.
08: Additions to what was qmp_block_dirty_enable/disable
    are no longer present as those function no longer exist.
09: Since 2.4
10: Since 2.4
    Demingled QMP command documentation.
11: Since 2.4
15: Test 112 --> 124
17: Number of tests altered. (Only 4, now.)

Fam Zheng (1):
  qapi: Add optional field "name" to block dirty bitmap

John Snow (20):
  docs: incremental backup documentation
  qmp: Ensure consistent granularity type
  qmp: Add block-dirty-bitmap-add and block-dirty-bitmap-remove
  block: Introduce bdrv_dirty_bitmap_granularity()
  hbitmap: cache array lengths
  hbitmap: add hbitmap_merge
  block: Add bitmap disabled status
  block: Add bitmap successors
  qmp: Add support of "dirty-bitmap" sync mode for drive-backup
  qmp: add block-dirty-bitmap-clear
  qmp: Add dirty bitmap status field in query-block
  block: add BdrvDirtyBitmap documentation
  block: Ensure consistent bitmap function prototypes
  block: Resize bitmaps on bdrv_truncate
  hbitmap: truncate tests
  iotests: add invalid input incremental backup tests
  iotests: add QMP event waiting queue
  iotests: add simple incremental backup case
  iotests: add incremental backup failure recovery test
  iotests: add incremental backup granularity tests

 block.c                       | 243 ++++++++++++++++++++++++++--
 block/backup.c                | 155 +++++++++++++++---
 block/mirror.c                |  46 +++---
 blockdev.c                    | 176 +++++++++++++++++++-
 docs/bitmaps.md               | 352 ++++++++++++++++++++++++++++++++++++++++
 hmp.c                         |   3 +-
 include/block/block.h         |  33 +++-
 include/block/block_int.h     |   4 +-
 include/qemu/hbitmap.h        |  23 +++
 migration/block.c             |   9 +-
 qapi/block-core.json          |  91 ++++++++++-
 qmp-commands.hx               |  93 ++++++++++-
 scripts/qmp/qmp.py            |  95 +++++++----
 tests/qemu-iotests/124        | 363 ++++++++++++++++++++++++++++++++++++++++++
 tests/qemu-iotests/124.out    |   5 +
 tests/qemu-iotests/group      |   1 +
 tests/qemu-iotests/iotests.py |  38 +++++
 tests/test-hbitmap.c          | 255 +++++++++++++++++++++++++++++
 util/hbitmap.c                |  85 ++++++++++
 19 files changed, 1953 insertions(+), 117 deletions(-)
 create mode 100644 docs/bitmaps.md
 create mode 100644 tests/qemu-iotests/124
 create mode 100644 tests/qemu-iotests/124.out

-- 
2.1.0