[Qemu-devel] [PATCH v4 RFC] spec: add qcow2-dirty-bitmaps specification

[Vladimir Sementsov-Ogievskiy]

The new feature for qcow2: storing dirty bitmaps.

Only dirty bitmaps relative to this qcow2 image should be stored in it.

Strings started from +# are RFC-strings, not to be commited of course.


Signed-off-by: Vladimir Sementsov-Ogievskiy <address@hidden>
---

 docs/specs/qcow2.txt | 151 ++++++++++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 150 insertions(+), 1 deletion(-)

diff --git a/docs/specs/qcow2.txt b/docs/specs/qcow2.txt
index 121dfc8..3c89580 100644
--- a/docs/specs/qcow2.txt
+++ b/docs/specs/qcow2.txt
@@ -103,7 +103,17 @@ in the description of a field.
                     write to an image with unknown auto-clear features if it
                     clears the respective bits from this field first.
 
-                    Bits 0-63:  Reserved (set to 0)
+                    Bit 0:      Dirty bitmaps bit.
+                                This bit is responsible for Dirty bitmaps
+                                extension consistency.
+                                If it is set, but there is no Dirty bitmaps
+                                extensions, this should be considered as an
+                                error.
+                                If it is not set, but there is a Dirty bitmaps
+                                extension, its data should be considered as
+                                inconsistent.
+
+                    Bits 1-63:  Reserved (set to 0)
 
          96 -  99:  refcount_order
                     Describes the width of a reference count block entry (width
@@ -123,6 +133,7 @@ be stored. Each extension has a structure like the 
following:
                         0x00000000 - End of the header extension area
                         0xE2792ACA - Backing file format name
                         0x6803f857 - Feature name table
+                        0x23852875 - Dirty bitmaps
                         other      - Unknown header extension, can be safely
                                      ignored
 
@@ -166,6 +177,31 @@ the header extension data. Each entry look like this:
                     terminated if it has full length)
 
 
+== Dirty bitmaps ==
+
+Dirty bitmaps is an optional header extension. It provides an ability to store
+dirty bitmaps in a qcow2 image. The data of this extension should be considered
+as consistent only if corresponding auto-clear feature bit is set (see
+autoclear_features above).
+The fields of Dirty bitmaps extension are:
+
+          0 -  3:  nb_dirty_bitmaps
+                   The number of dirty bitmaps contained in the image. Valid
+                   values: 1 - 65535.
+# Let's be strict, the feature should be deleted with deleting last bitmap.
+
+          4 -  7:  dirty_bitmap_directory_size
+                   Size of the Dirty Bitmap Directory in bytes. It should be
+                   equal to sum of sizes of all (nb_dirty_bitmaps) dirty bitmap
+                   headers.
+# This field is necessary to effectively read Dirty Bitmap Directory, because
+# it's entries (which are dirty bitmap headers) may have different lengths.
+
+          8 - 15:  dirty_bitmap_directory_offset
+                   Offset into the image file at which the Dirty Bitmap
+                   Directory starts. Must be aligned to a cluster boundary.
+
+
 == Host cluster management ==
 
 qcow2 manages the allocation of host clusters by maintaining a reference count
@@ -360,3 +396,116 @@ Snapshot table entry:
 
         variable:   Padding to round up the snapshot table entry size to the
                     next multiple of 8.
+
+
+== Dirty bitmaps ==
+
+The feature supports storing dirty bitmaps in a qcow2 image. All dirty bitmaps
+are relating to the virtual disk, stored in this image.
+
+=== Dirty Bitmap Directory ===
+
+Each dirty bitmap saved in the image is described in a Dirty Bitmap Directory
+entry. Dirty Bitmap Directory is a contiguous area in the image file, whose
+starting offset and length are given by the header extension fields
+dirty_bitmap_directory_offset and dirty_bitmap_directory_size. The entries of
+the bitmap directory have variable length, depending on the length of the
+bitmap name. These entries are also called dirty bitmap headers.
+
+Dirty Bitmap Directory Entry:
+
+    Byte 0 -  7:    dirty_bitmap_table_offset
+                    Offset into the image file at which the Dirty Bitmap Table
+                    (described below) for the bitmap starts. Must be aligned to
+                    a cluster boundary.
+
+         8 - 11:    dirty_bitmap_table_size
+                    Number of entries in the Dirty Bitmap Table of the bitmap.
+
+        12 - 15:    flags
+                    Bit
+                      0: in_use
+                         The bitmap was not saved correctly and may be
+                         inconsistent.
+
+                      1: auto
+                         The bitmap should be autoloaded as block dirty bitmap
+                         and tracking should be started. Type of the bitmap
+                         should be 'Dirty Tracking Bitmap'.
+
+                    Bits 2 - 31 are reserved and must be 0.
+
+        16 - 17:    name_size
+                    Size of the bitmap name. Valid values: 1 - 1023.
+
+             18:    type
+                    This field describes the sort of the bitmap.
+                    Values:
+                      0: Dirty Tracking Bitmap
+
+                    Values 1 - 255 are reserved.
+# This is mostly for error checking and information in qemu-img info output.
+# The other types may be, for example, "Backup Bitmap" - to make it possible
+# stop backup job on vm stop and resume it later. The another one is "Sector
+# Alloction Bitmap" (Fam, John, please comment).
+
+             19:    granularity_bits
+                    Granularity bits. Valid values are: 0 - 31.
+# Now, qemu allows creating bitmaps with granularity as a 32bit value. And
+# there are no reasons of increasing it.
+
+                    Granularity is calculated as
+                        granularity = 1 << granularity_bits
+
+                    Granularity of the bitmap is how many bytes of the image
+                    accounts for one bit of the bitmap.
+# To be closer to qcow2 and its reality, I've decided to use byte-granularity
+# here, not sector-granularity.
+
+        variable:   The name of the bitmap (not null terminated). Should be
+                    unique among all dirty bitmap names within the Dirty
+                    bitmaps extension.
+
+        variable:   Padding to round up the Dirty Bitmap Directory Entry size
+                    to the next multiple of 8.
+
+=== Dirty Bitmap Table ===
+
+Dirty bitmaps are stored using a one-level (not two-level like refcounts and
+guest clusters mapping) structure for the mapping of bitmaps to host clusters.
+It is called Dirty Bitmap Table.
+
+Each Dirty Bitmap Table has a variable size (stored in the Dirty Bitmap
+Directory Entry) and may use multiple clusters, however it must be contiguous
+in the image file.
+
+Given an offset (in bytes) into the bitmap, the offset into the image file can
+be obtained as follows:
+
+    byte_offset =
+        dirty_bitmap_table[offset / cluster_size] + (offset % cluster_size)
+
+Taking into account the granularity of the bitmap, an offset in bits into the
+image file, corresponding to byte number byte_nr of the image can be calculated
+like this:
+
+    bit_offset =
+        byte_offset(byte_nr / granularity / 8) * 8 + (byte_nr / granularity) % 
8
+
+Note: the last formula is only for understanding the things, it is unlikely for
+it to be useful in a program.
+
+Dirty Bitmap Table entry:
+
+    Bit       0:    Reserved and should be zero if bits 9 - 55 are non-zero.
+                    If bits 9 - 55 are zero:
+                      0: Cluster should be read as all zeros.
+                      1: Cluster should be read as all ones.
+
+         1 -  8:    Reserved and must be zero.
+
+         9 - 55:    Bits 9 - 55 of host cluster offset. Must be aligned to a
+                    cluster boundary. If the offset is 0, the cluster is
+                    unallocated, see bit 0 description.
+
+        56 - 63:    Reserved, must be 0.
-- 
2.1.4