[Qemu-devel] [PATCH v3 00/22] qcow2: Support refcount orders != 4

[Max Reitz]

As of version 3, the qcow2 file format supports different widths for
refcount entries, ranging from 1 to 64 bit (only powers of two).
Currently, qemu only supports 16 bit, which is the only width supported
by version 2 (compat=0.10) images.

This series adds support to qemu for all other valid refcount orders.
This is mainly done by adding two function pointers into the
BDRVQcowState structure for reading and writing refcount values
independently of the current refcount entry width; all in-memory
refcount arrays (mostly cached refcount blocks) now are void pointers
and are accessed through these functions alone.

Thanks to previous work of making the qemu code agnostic of e.g. the
number of refcount entries per refcount block, the rest is fairly
trivial. The most complex patch in this series is patch 18 which
implements changing the refcount order through qemu-img amend.

To test different refcount widths, simply invoke the qemu-iotests check
program with -o refcount_width=${your_desired_width}. The final test in
this series adds some tests for operations which do not work with
certain refcount orders and for refcount order amendment.


This series depends on version 4 (or any later version) of my
"chardev: Add -qmp-pretty" series (due to different test output of test
067, which makes changing it here much nicer).


v3 (everything [Eric], except for patch 13):
- Patch 6:
  - s/independently/independent/ in the commit message
  - Extended the commit message to tell why the new helper function
    always aligns the size of the refcount array on cluster boundaries
  - Added a comment to realloc_refcount_array() (the new helper
    function) which explains what this function does and why it does it
    the way it does it
  - Immediately return from realloc_refcount_array() if the size of the
    refcount array decreases (should not happen, but doesn't hurt
    either) or stays the same (pretty likely); this prevents us from
    going into g_try_realloc() with a size of 0, which in turn
    guarantees that it will never return a NULL pointer on success
- Patch 7:
  - Extended the commit message to explain why the bounce buffer for
    refblocks in realloc_refcount_array() can be omitted and that it
    will therefore be omitted
  - Fixed alignment of check_refcounts_l2()'s header
  - Use refcount_array_byte_size() when clearing the refcount array
    before recalculating the refcounts
- Patch 8:
  - Use two look-up tables for obtaining get_refcount() and
    set_refcount(), respectively, instead of a single switch statement
    (it is indeed much shorted and better readable this way)
- Patch 9: s/bit/bits/
- Patch 10:
  - Extended the commit message to note changes to the preallocation
    file size calculation
  - Added a comment to that calculation that it does not need to be
    exact (this addition is noted in the commit message as well)
- Patch 11: Added a missing comment
- Patch 13: Added
- Patch 19 (prev. 18):
  - get_refcount_functions() has been replaced by two look-up tables
    (patch 8)
  - s/MIN/MAX/
  - Dropped two dead new_allocation assignments
  - s/QCOW2_DISCARD_NEVER/QCOW2_DISCARD_OTHER/ in two places
- Patch 22 (prev. 21):
  - Added a test case for the MSb set for 64 bit refcounts
  - Extended the 3-pass amend test case to test whether there are
    actually three passes (by grepping in the progress report output)


git-backport-diff against v2:

Key:
[----] : patches are identical
[####] : number of functional differences between upstream/downstream patch
[down] : patch is downstream-only
The flags [FC] indicate (F)unctional and (C)ontextual differences, respectively

001/22:[----] [--] 'qcow2: Add two new fields to BDRVQcowState'
002/22:[----] [--] 'qcow2: Add refcount_width to format-specific info'
003/22:[----] [--] 'qcow2: Use 64 bits for refcount values'
004/22:[----] [--] 'qcow2: Respect error in qcow2_alloc_bytes()'
005/22:[----] [--] 'qcow2: Refcount overflow and qcow2_alloc_bytes()'
006/22:[0024] [FC] 'qcow2: Helper for refcount array reallocation'
007/22:[0007] [FC] 'qcow2: Helper function for refcount modification'
008/22:[0068] [FC] 'qcow2: More helpers for refcount modification'
009/22:[0002] [FC] 'qcow2: Open images with refcount order != 4'
010/22:[0005] [FC] 'qcow2: refcount_order parameter for qcow2_create2'
011/22:[0001] [FC] 'iotests: Prepare for refcount_width option'
012/22:[----] [--] 'qcow2: Allow creation with refcount order != 4'
013/22:[down] 'progress: Allow regressing progress'
014/22:[----] [--] 'block: Add opaque value to the amend CB'
015/22:[----] [--] 'qcow2: Use error_report() in qcow2_amend_options()'
016/22:[----] [--] 'qcow2: Use abort() instead of assert(false)'
017/22:[----] [--] 'qcow2: Split upgrade/downgrade paths for amend'
018/22:[----] [--] 'qcow2: Use intermediate helper CB for amend'
019/22:[0013] [FC] 'qcow2: Add function for refcount order amendment'
020/22:[----] [--] 'qcow2: Invoke refcount order amendment function'
021/22:[----] [--] 'qcow2: Point to amend function in check'
022/22:[0052] [FC] 'iotests: Add test for different refcount widths'


Max Reitz (22):
  qcow2: Add two new fields to BDRVQcowState
  qcow2: Add refcount_width to format-specific info
  qcow2: Use 64 bits for refcount values
  qcow2: Respect error in qcow2_alloc_bytes()
  qcow2: Refcount overflow and qcow2_alloc_bytes()
  qcow2: Helper for refcount array reallocation
  qcow2: Helper function for refcount modification
  qcow2: More helpers for refcount modification
  qcow2: Open images with refcount order != 4
  qcow2: refcount_order parameter for qcow2_create2
  iotests: Prepare for refcount_width option
  qcow2: Allow creation with refcount order != 4
  progress: Allow regressing progress
  block: Add opaque value to the amend CB
  qcow2: Use error_report() in qcow2_amend_options()
  qcow2: Use abort() instead of assert(false)
  qcow2: Split upgrade/downgrade paths for amend
  qcow2: Use intermediate helper CB for amend
  qcow2: Add function for refcount order amendment
  qcow2: Invoke refcount order amendment function
  qcow2: Point to amend function in check
  iotests: Add test for different refcount widths

 block.c                          |   4 +-
 block/qcow2-cluster.c            |  23 +-
 block/qcow2-refcount.c           | 938 +++++++++++++++++++++++++++++++++------
 block/qcow2.c                    | 261 ++++++++---
 block/qcow2.h                    |  24 +-
 include/block/block.h            |   4 +-
 include/block/block_int.h        |   4 +-
 qapi/block-core.json             |   5 +-
 qemu-img.c                       |   5 +-
 tests/qemu-iotests/007           |   3 +
 tests/qemu-iotests/015           |   2 +
 tests/qemu-iotests/026           |   7 +
 tests/qemu-iotests/029           |   2 +
 tests/qemu-iotests/049.out       | 112 ++---
 tests/qemu-iotests/051           |   3 +
 tests/qemu-iotests/058           |   2 +
 tests/qemu-iotests/060.out       |   1 +
 tests/qemu-iotests/061.out       |  14 +-
 tests/qemu-iotests/065           |  23 +-
 tests/qemu-iotests/067           |   2 +
 tests/qemu-iotests/067.out       |   5 +
 tests/qemu-iotests/079           |  10 +-
 tests/qemu-iotests/079.out       |  38 +-
 tests/qemu-iotests/080           |   2 +
 tests/qemu-iotests/082.out       |  48 +-
 tests/qemu-iotests/085.out       |  38 +-
 tests/qemu-iotests/089           |   2 +
 tests/qemu-iotests/089.out       |   2 +
 tests/qemu-iotests/108           |   2 +
 tests/qemu-iotests/112           | 278 ++++++++++++
 tests/qemu-iotests/112.out       | 143 ++++++
 tests/qemu-iotests/common.filter |   3 +-
 tests/qemu-iotests/group         |   1 +
 util/qemu-progress.c             |   1 +
 34 files changed, 1680 insertions(+), 332 deletions(-)
 create mode 100755 tests/qemu-iotests/112
 create mode 100644 tests/qemu-iotests/112.out

-- 
1.9.3