[Qemu-devel] [RFC v2 00/15] QED image streaming

[Stefan Hajnoczi]

Overview
--------

This patch series adds image streaming support for QED image files.  QMP/HMP
commands are added to perform image streaming at runtime.  This interface is
already supported by libvirt.

The goal is to implement image streaming in a generic way for all image formats
that support backing files.  In the meantime, I want to share the latest
QED-specific patch series.

Image streaming populates the file in the background while the guest is
running.  This makes it possible to start the guest before its image file has
been fully provisioned.

Example use cases include:
 * Providing small virtual appliances for download that can be launched
  immediately but provision themselves in the background.
 * Reducing guest provisioning time by creating local image files but backing
  them with shared master images which will be streamed.

When image streaming is enabled, the unallocated regions of the image file are
populated with the data from the backing file.  This occurs in the background
and the guest can perform regular I/O in the meantime.  Once the entire backing
file has been streamed, the image no longer requires a backing file and will
drop its reference.

Example invocation
------------------

$ # my_fedora.qed is a tiny file initially but will be streamed when the guest 
starts
$ ./qemu-img create -f qed -o backing_file=fedora-14.img my_fedora.qed
Formatting 'my_fedora.qed', fmt=qed size=10737418240 
backing_file='fedora-14.img' cluster_size=0 table_size=0

$ # run the guest and stream fedora-14.img into my_fedora.qed
$ x86_64-softmmu/qemu-system-x86_64 -m 512 -enable-kvm -drive 
if=virtio,file=my_fedora.qed,cache=none,stream=on

Details on changes
------------------

Image streaming introduces a new bdrv_aio_copy_backing() interface.  Block
drivers that implement this interface support streaming.  This function scans
for an unallocated cluster and populates it with data from the backing file.

The details of populating the image file are actually best implemented as a
copy-on-read operation.  Copy-on-read means that a read request will populate
the image file if it needs to fetch data from the backing file.  The
copy-on-read feature can be used outside the context of streaming and this
patch series therefore introduces the -drive copy-on-read=on option for that
purpose.

The new block_stream QMP/HMP command can be used to start streaming a block
device.  QMP events are raised on completion and failure so that polling is not
required.  Adam Litke <address@hidden> has implemented the libvirt APIs for
image streaming:
http://www.redhat.com/archives/libvir-list/2011-July/msg01570.html

Patches 1-6
block: add -drive copy-on-read=on|off
qed: replace is_write with flags field
qed: extract qed_start_allocating_write()
qed: make qed_aio_write_alloc() reusable
qed: add support for copy-on-read
qed: avoid deadlock on emulated synchronous I/O

  These patches add copy-on-read support and implement it for QED.

Patches 7-13
block: add bdrv_aio_copy_backing()
qmp: add block_stream command
qmp: add block_job_cancel command
qmp: add query-block-jobs command
qmp: add block_job_set_speed command
block: add -drive stream=on|off
qed: intelligent streaming implementation

  These patches implement image streaming using copy-on-read.

Patch 14
trace: trace bdrv_aio_readv/writev error paths

  Additional trace events to identify I/O errors.

Patch 15
tests: add image streaming QMP interface tests

  A Python script that performs basic QMP tests of image streaming.

v2:
 * Implement latest block_stream QMP/HMP API
 * Split monitor command patches into separate commits
 * Add rate-limiting
 * Remove iteration interface where client drives streaming

v1:
 * -drive copy-on-read=on|off,stream=on|off instead of image header bits
 * Latest libvirt API compatibility
 * Workaround and assert for synchronous I/O emulation deadlock

Anthony Liguori (3):
  qed: add support for copy-on-read
  block: add bdrv_aio_copy_backing()
  qed: intelligent streaming implementation

Stefan Hajnoczi (12):
  block: add -drive copy-on-read=on|off
  qed: replace is_write with flags field
  qed: extract qed_start_allocating_write()
  qed: make qed_aio_write_alloc() reusable
  qed: avoid deadlock on emulated synchronous I/O
  qmp: add block_stream command
  qmp: add block_job_cancel command
  qmp: add query-block-jobs command
  qmp: add block_job_set_speed command
  block: add -drive stream=on|off
  trace: trace bdrv_aio_readv/writev error paths
  tests: add image streaming QMP interface tests

 block.c         |   66 +++++++++-
 block.h         |    6 +
 block/qed.c     |  363 ++++++++++++++++++++++++++++++++++++++++++++++--------
 block/qed.h     |    8 +-
 block_int.h     |    3 +
 blockdev.c      |  286 +++++++++++++++++++++++++++++++++++++++++++
 blockdev.h      |    8 ++
 hmp-commands.hx |   51 ++++++++-
 monitor.c       |   19 +++
 monitor.h       |    1 +
 qemu-config.c   |    8 ++
 qemu-options.hx |   13 ++-
 qerror.c        |    8 ++
 qerror.h        |    6 +
 qmp-commands.hx |  171 ++++++++++++++++++++++++++
 test-stream.py  |  193 +++++++++++++++++++++++++++++
 trace-events    |   10 ++-
 17 files changed, 1156 insertions(+), 64 deletions(-)
 create mode 100644 test-stream.py

-- 
1.7.5.4