[Qemu-devel] [PATCH v6 00/39] blockdev: BlockBackend and media

[Max Reitz]

*** This series is based on v3 of Kevin's  ***
*** "block: Get rid of bdrv_swap()" series ***

This series reworks a lot regarding BlockBackend and media. Basically,
it allows empty BlockBackends, that is BBs without a BDS tree.

Before this series, empty drives are represented by a BlockBackend with
an empty BDS attached to it (a BDS with a NULL driver). However, now we
have BlockBackends, thus an empty drive should be represented by a
BlockBackend without any BDS tree attached to it. This is what this
series does.


Quick and early summary for the v6 changes:
- Rebase on master and Kevin's bdrv_swap() series
- Addressed Kevin's comments for v5


Justification for each of the patches and their order:

-- Preparation before _is_inserted() patches --
 1: Patch 9 will not take care not to break host floppy support, so that
    support needs to be removed first.
 2: Needed for patch 3, so that blockdev-added BDSs without a BB still
    get the BDRV_O_INCOMING flag set.
 3: Needed for patch 4. Patch 26 is a follow-up after BDS-less BBs are
    allowed.
 4: bdrv_close_all() is broken ("block: Rework bdrv_close_all()"). Patch
    7 will break iotest 071 (actually, just make the problem apparent).
    So this patch is required to work around the issue.
    (with "the issue" being that bdrv_close_all() does not unref() the
    BDSs it is closing, but just force-closes everything, even if the
    BDS may still be in use somewhere)

-- _is_inserted() patches --
 5: General clean-up work, nice to have before patch 7 (and goes in tune
    with patch 6).
 6: Using the same BB as a guest device means that the data read from
    there should be exactly the same. Opening the guest tray should
    therefore result in no data being readable. This is what we then
    need this function for.
 7: General clean-up work (in the _is_inserted() area).
 8: General clean-up work (in the _is_inserted() area).
 9: General clean-up work (also regarding _is_inserted()).
10: Required so inserting a floppy will not result in the tray being
    reported as closed (you need to "push in" the floppy first, using
    blockdev-close-tray). It's here in the "_is_inserted() patches area"
    because I feel like that's a very related topic.

-- Support for BDS-less BBs --
11: Preparation for BDS-less BBs
12: Preparation for BDS-less BBs
13: Preparation for BDS-less BBs (BB properties should be in the BB, and
    not in the root BDS)
14: Patch 15 removes BlockAcctStats from the BDS, but wr_highest_sector
    is BDS-dependent, so it needs to stay here
15: Preparation for BDS-less BBs (BB properties should be in the BB, and
    not in the root BDS)
16: Preparation for BDS-less BBs (BB properties should be in the BB, and
    not in the root BDS)
17: Needed for patch 18
18: Preparation for BDS-less BBs (Removing a BDS tree should retain some
    properties for legacy reasons, which must therefore be stored in the
    BB(RS))
19: Preparation for BDS-less BBs
20: Preparation for BDS-less BBs
21: Preparation for BDS-less BBs
22: Ability to add BDS trees to empty BBs ("inserting a medium")
23: Preparation for BDS-less BBs (needs patch 22)
24: One goal of this series, and fixes the "opening tray" event for
    empty drives when shutting down qemu
25: Needed for patch 26
26: Completion of what patch 3 begun
27: Ability to detach BDS trees from BBs

-- "Atomic" QMP tray operations --
28: blockdev-open-tray
29: blockdev-close-tray
30: blockdev-remove-medium
31: blockdev-insert-medium

-- Reimplementation of change/eject --
32: eject
33: change
34: Clean-up patch

-- New QMP blockdev-change-medium command --
35: New QMP command
36: Use for HMP change command
37: Add flag to that command for changing the read-only access mode
    (which was my original intention for this series)
38: Same flag for HMP

-- Tests --
39: iotests are always nice, so here is one


v6:
- Patch 8: Trivial rebase conflict due to Kevin's bdrv_swap() series
- Patch 17: Added, so that the throttle group can be strongly referenced
  by the BBRS
- Patch 18:
  - Keep a strong reference to the throttle group in the BBRS [Kevin];
    this is done by keeping the name of the group (which the external
    interface of the throttle implementation generally uses for
    identification) and a pointer to the ThrottleState. Just storing the
    name is not enough, we actually do need the ThrottleState for patch
    24.
    The ThrottleConfig is dropped since the ThrottleGroup's
    configuration should not be overwritten when a new BDS is added.
- Patch 19: blk_{set_,}enable_write_cache() can fall back to the BBRS,
  too [Kevin]
- Patch 21: Added blk_drain(), removed blk_{set_,}enable_write_cache()
  [Kevin]
- Patch 22: blk_insert_bs() is no longer idempotent [Kevin, Eric]
  (i.e., inserting a BDS into its attached BB will now fail an
  assertion)
- Patch 24:
  - Adopted to how the BBRS stores the ThrottleGroup reference now. An
    important point is that if a BDS tree is created,
    bdrv_set_io_limits() is called after bdrv_io_limits_enable().
    Therefore, when creating an empty BB but assigning it to a throttle
    group, we have to change the group's configuration, too. This is why
    we need the ThrottleState pointer in the BBRS, so we can change a
    throttle group's configuration without knowing a BDS which is part
    of the group.
- Patch 27:
  - blk_remove_bs() is no longer idempotent (i.e. removing a BDS from a
    BB without an attached BDS will now fail an assertion)
  - blk_insert_bs() using a BDS that is already attached to a BB will no
    longer automatically detach that BDS from its BB, but will now fail
    an assertion
- Patch 28: s/qmp_marshal_input_/qmp_marshal_/
- Patch 29: s/qmp_marshal_input_/qmp_marshal_/
- Patch 30: s/qmp_marshal_input_/qmp_marshal_/
- Patch 31:
  - s/qmp_marshal_input_/qmp_marshal_/
  - Following the changes to patch 27, trying to insert a BDS tree that
    already has a BB into another BB is now an error
    (silently detaching the old BB was a bad idea)
- Patch 33: The new BDS is now just added to the throttle group thus
  setting its throttling configuration. bdrv_set_io_limits() is no
  longer called (this would reset the group's configuration). [Kevin]
- Patch 35: s/qmp_marshal_input_/qmp_marshal_/


git-backport-diff against v5:

Key:
[----] : patches are identical
[####] : number of functional differences between upstream/downstream patch
[down] : patch is downstream-only
The flags [FC] indicate (F)unctional and (C)ontextual differences, respectively

001/39:[----] [--] 'block: Remove host floppy support'
002/39:[----] [--] 'block: Set BDRV_O_INCOMING in bdrv_fill_options()'
003/39:[----] [--] 'blockdev: Allow creation of BDS trees without BB'
004/39:[----] [--] 'iotests: Only create BB if necessary'
005/39:[----] [-C] 'block: Make bdrv_is_inserted() return a bool'
006/39:[----] [--] 'block: Add blk_is_available()'
007/39:[----] [--] 'block: Make bdrv_is_inserted() recursive'
008/39:[0002] [FC] 'block/raw_bsd: Drop raw_is_inserted()'
009/39:[----] [-C] 'block: Invoke change media CB before NULLing drv'
010/39:[----] [--] 'hw/block/fdc: Implement tray status'
011/39:[----] [--] 'hw/usb-storage: Check whether BB is inserted'
012/39:[----] [--] 'block: Fix BB AIOCB AioContext without BDS'
013/39:[----] [-C] 'block: Move guest_block_size into BlockBackend'
014/39:[----] [-C] 'block: Remove wr_highest_sector from BlockAcctStats'
015/39:[----] [--] 'block: Move BlockAcctStats into BlockBackend'
016/39:[----] [-C] 'block: Move I/O status and error actions into BB'
017/39:[down] 'block/throttle-groups: Make incref/decref public'
018/39:[0029] [FC] 'block: Add BlockBackendRootState'
019/39:[0016] [FC] 'block: Make some BB functions fall back to BBRS'
020/39:[----] [--] 'block: Fail requests to empty BlockBackend'
021/39:[0010] [FC] 'block: Prepare remaining BB functions for NULL BDS'
022/39:[0007] [FC] 'block: Add blk_insert_bs()'
023/39:[----] [--] 'block: Prepare for NULL BDS'
024/39:[0006] [FC] 'blockdev: Do not create BDS for empty drive'
025/39:[----] [--] 'blockdev: Pull out blockdev option extraction'
026/39:[----] [--] 'blockdev: Allow more options for BB-less BDS tree'
027/39:[0010] [FC] 'block: Add blk_remove_bs()'
028/39:[0002] [FC] 'blockdev: Add blockdev-open-tray'
029/39:[0002] [FC] 'blockdev: Add blockdev-close-tray'
030/39:[0002] [FC] 'blockdev: Add blockdev-remove-medium'
031/39:[0008] [FC] 'blockdev: Add blockdev-insert-medium'
032/39:[----] [--] 'blockdev: Implement eject with basic operations'
033/39:[0005] [FC] 'blockdev: Implement change with basic operations'
034/39:[----] [--] 'block: Inquire tray state before tray-moved events'
035/39:[0002] [FC] 'qmp: Introduce blockdev-change-medium'
036/39:[----] [--] 'hmp: Use blockdev-change-medium for change command'
037/39:[----] [-C] 'blockdev: read-only-mode for blockdev-change-medium'
038/39:[----] [--] 'hmp: Add read-only-mode option to change command'
039/39:[----] [--] 'iotests: Add test for change-related QMP commands'



Max Reitz (39):
  block: Remove host floppy support
  block: Set BDRV_O_INCOMING in bdrv_fill_options()
  blockdev: Allow creation of BDS trees without BB
  iotests: Only create BB if necessary
  block: Make bdrv_is_inserted() return a bool
  block: Add blk_is_available()
  block: Make bdrv_is_inserted() recursive
  block/raw_bsd: Drop raw_is_inserted()
  block: Invoke change media CB before NULLing drv
  hw/block/fdc: Implement tray status
  hw/usb-storage: Check whether BB is inserted
  block: Fix BB AIOCB AioContext without BDS
  block: Move guest_block_size into BlockBackend
  block: Remove wr_highest_sector from BlockAcctStats
  block: Move BlockAcctStats into BlockBackend
  block: Move I/O status and error actions into BB
  block/throttle-groups: Make incref/decref public
  block: Add BlockBackendRootState
  block: Make some BB functions fall back to BBRS
  block: Fail requests to empty BlockBackend
  block: Prepare remaining BB functions for NULL BDS
  block: Add blk_insert_bs()
  block: Prepare for NULL BDS
  blockdev: Do not create BDS for empty drive
  blockdev: Pull out blockdev option extraction
  blockdev: Allow more options for BB-less BDS tree
  block: Add blk_remove_bs()
  blockdev: Add blockdev-open-tray
  blockdev: Add blockdev-close-tray
  blockdev: Add blockdev-remove-medium
  blockdev: Add blockdev-insert-medium
  blockdev: Implement eject with basic operations
  blockdev: Implement change with basic operations
  block: Inquire tray state before tray-moved events
  qmp: Introduce blockdev-change-medium
  hmp: Use blockdev-change-medium for change command
  blockdev: read-only-mode for blockdev-change-medium
  hmp: Add read-only-mode option to change command
  iotests: Add test for change-related QMP commands

 block.c                         | 180 ++------
 block/accounting.c              |   8 -
 block/backup.c                  |  17 +-
 block/block-backend.c           | 389 +++++++++++++++--
 block/commit.c                  |   3 +-
 block/io.c                      |  10 +-
 block/mirror.c                  |  17 +-
 block/qapi.c                    |  36 +-
 block/raw-posix.c               | 230 +---------
 block/raw_bsd.c                 |   6 -
 block/stream.c                  |   3 +-
 block/throttle-groups.c         |  19 +-
 blockdev.c                      | 921 +++++++++++++++++++++++++++++-----------
 blockjob.c                      |   5 +-
 hmp-commands.hx                 |  20 +-
 hmp.c                           |  47 +-
 hw/block/fdc.c                  |  20 +-
 hw/block/xen_disk.c             |   4 +-
 hw/usb/dev-storage.c            |  30 +-
 include/block/accounting.h      |   3 -
 include/block/block.h           |  16 +-
 include/block/block_int.h       |  25 +-
 include/block/throttle-groups.h |   3 +
 include/qemu/typedefs.h         |   1 +
 include/sysemu/block-backend.h  |  14 +-
 include/sysemu/blockdev.h       |   2 -
 migration/block.c               |   5 +
 monitor.c                       |   4 +
 qapi-schema.json                |   6 +-
 qapi/block-core.json            | 138 +++++-
 qmp-commands.hx                 | 213 +++++++++-
 qmp.c                           |   9 +-
 tests/fdc-test.c                |   4 +-
 tests/qemu-iotests/071          |  54 ++-
 tests/qemu-iotests/071.out      |  12 +-
 tests/qemu-iotests/081          |  18 +-
 tests/qemu-iotests/081.out      |   5 +-
 tests/qemu-iotests/087          |   2 +-
 tests/qemu-iotests/087.out      |   4 +-
 tests/qemu-iotests/118          | 638 ++++++++++++++++++++++++++++
 tests/qemu-iotests/118.out      |   5 +
 tests/qemu-iotests/group        |   1 +
 42 files changed, 2313 insertions(+), 834 deletions(-)
 create mode 100755 tests/qemu-iotests/118
 create mode 100644 tests/qemu-iotests/118.out

-- 
2.6.1