[Qemu-devel] [RFC V8 00/24] QCOW2 deduplication core functionality

[Benoît Canet]

Hello,

This is the new patchset of the QCOW2 deduplication feature using an on disk
key value store modelled after the SILT one.

This is more a millestone reached post than a for review post.

Performance is slow because the code is issuing a lot of read and write to the
SSD in a sequential ways.
The next revision will focus on parallelising the IOs.

Kevin: the first patch add the description of the QCOW2 journal structure.
       The code make it easy to have multiple journal in QCOW2.
       Do you think it could fit others QCOW2 usages.
       Patches describing the others data structure are comming.

Best regards

Benoît

Benoît Canet (24):
  qcow2: Add journal specification.
  qcow2: Add deduplication structures and fields.
  qcow2: Add journal.
  qcow2: Create the log store.
  qcow2: Add the hash store.
  qcow2: Add the deduplication store.
  qcow2: Add qcow2_dedup_read_missing_and_concatenate
  qcow2: Create a way to link to l2 tables when deduplicating.
  qcow2: Make qcow2_update_cluster_refcount public.
  qcow2: Add qcow2_dedup and related functions
  qcow2: Add qcow2_dedup_store_new_hashes.
  qcow2: Do allocate on rewrite on the dedup case.
  qcow2: Implement qcow2_compute_cluster_hash.
  qcow2: Load and save deduplication table header extension.
  qcow2: Extract qcow2_set_incompat_feature and
    qcow2_clear_incompat_feature.
  block: Add qcow2_dedup format and image creation code.
  qcow2: Drop hash for a given cluster when dedup makes refcount >
    2^16/2.
  qcow2: Remove hash when cluster is deleted.
  qcow2: Integrate deduplication in qcow2_co_writev loop.
  qcow2: Serialize write requests when deduplication is activated.
  qcow2: Integrate SKEIN hash algorithm in deduplication.
  qcow2: Add qcow2_dedup_init and qcow2_dedup_close.
  qcow2: Enable the deduplication feature.
  qcow2: Enable deduplication tests

 block/Makefile.objs          |    1 +
 block/qcow2-cluster.c        |   13 +-
 block/qcow2-dedup.c          |  778 +++++++++++++++++++++++++
 block/qcow2-hash-store.c     |  802 ++++++++++++++++++++++++++
 block/qcow2-journal.c        |  587 +++++++++++++++++++
 block/qcow2-log-store.c      | 1281 ++++++++++++++++++++++++++++++++++++++++++
 block/qcow2-refcount.c       |   20 +-
 block/qcow2-store.c          |  771 +++++++++++++++++++++++++
 block/qcow2.c                |  431 +++++++++++++-
 block/qcow2.h                |  357 +++++++++++-
 configure                    |  121 +++-
 docs/specs/qcow2.txt         |   42 ++
 include/block/block_int.h    |    1 +
 tests/qemu-iotests/common    |    6 +
 tests/qemu-iotests/common.rc |    3 +-
 15 files changed, 5144 insertions(+), 70 deletions(-)
 create mode 100644 block/qcow2-dedup.c
 create mode 100644 block/qcow2-hash-store.c
 create mode 100644 block/qcow2-journal.c
 create mode 100644 block/qcow2-log-store.c
 create mode 100644 block/qcow2-store.c

-- 
1.7.10.4