Re: [Qemu-block] [Qemu-devel] [RFC PATCH 01/16] doc: Add QBM format specification

[Markus Armbruster]

Eric Blake <address@hidden> writes:

> On 01/26/2016 03:38 AM, Fam Zheng wrote:
>> Signed-off-by: Fam Zheng <address@hidden>
>> ---
>>  docs/specs/qbm.md | 118 
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>  1 file changed, 118 insertions(+)
>>  create mode 100644 docs/specs/qbm.md
>> 
>> diff --git a/docs/specs/qbm.md b/docs/specs/qbm.md
>> new file mode 100644
>> index 0000000..b91910b
>> --- /dev/null
>> +++ b/docs/specs/qbm.md
>> @@ -0,0 +1,118 @@
>> +QEMU Block Bitmap (QBM)
>> +=======================
>
> No explicit copyright mention means that this document is GPLv2+ by
> default.  I don't know if any 3rd-party implementation trying to use
> this spec would object to that, or if a looser license is desirable here
> (I don't personally care, but just raising the point).

IANAL, but protecting the spec with a strong copyleft shouldn't preclude
differently licensed implementations.  It would preclude including the
spec in whatever they distribute.

>> +
>> +QBM is a multi-file disk format to allow storing persistent block bitmaps 
>> along
>> +with the tracked data image.  A QBM image includes one json descriptor file,
>> +one data image, one or more bitmap files that describe the block dirty 
>> status
>
> s/one or more/and one or more/
>
> Must it have block dirty status bitmaps, or can you have a QBM image
> with just an allocation bitmap?
>
>> +of the data image.
>> +
>> +The json file describes the structure of the image. The structure of the 
>> json
>> +descriptor file is:
>
> Please mention that the file must be valid JSON per RFC 7159.  Probably
> also worth requiring that the file is a text file (ends in a newline).
>
>> +
>> +    QBM-JSON-FILE := { "QBM": DESC-JSON }
>> +
>> +    DESC-JSON := { "version": 1,
>> +                   "image": IMAGE,
>> +                   "BITMAPS": BITMAPS
>
> s/"BITMAPS"/"bitmaps"/
>
> See my thoughts below on whether this is the ideal top-level structure.
>
>> +                 }
>> +
>> +Fields in the top level json dictionary are:
>> +
>> address@hidden: An integer which must be 1.
>> address@hidden: A dictionary in IMAGE schema, as described later. It 
>> provides the
>> +        information of the data image where user data is stored. Its format 
>> is
>> +        documented in the "IMAGE schema" section.
>> address@hidden: A dictionary that describes one ore more bitmap files. The 
>> keys into
>> +          the dictionary are the names of bitmap, which must be strings, and
>> +          each value is a dictionary describing the information of the 
>> bitmap,
>> +          as documented below in the "BITMAP schema" section.
>
> Making 'bitmaps' be a dictionary means that we now have keys that might
> not be an identifier.  Although this is valid JSON, it may trip up some
> tools.  Would it be better to make 'bitmaps' be a list of dictionaries,
> where each dictionary has a 'name':'value' member, so that bitmap names
> that are not a (C, Python, whatever) identifier are still valid?

We need to make up our mind whether to use JSON or the intersection of
JSON with other languages.  I'd stick to JSON and the more natural
schema Fam proposed.

>> +
>> +=== IMAGE schema ===
>> +
>> +An IMAGE records the information of an image (such as a data image or a 
>> backing
>> +file). It has following fields:
>
> I liked how you showed DESC-JSON := for the top level; should you do the
> same here for IMAGE?
>
>> +
>> address@hidden: The file name string of the referenced image. If it's a 
>> relative path,
>> +       the file should be found relative to the descriptor file's
>> +       location.
>
> Does that mean we'll have to use 'json:...' encoding for representing
> network resources?  Should we instead reuse some of the
> qapi/block-core.json representation of a block device?

This isn't QMP, where encoding data in strings is anathema, but the same
argument applies: instead of inventing ad hoc syntaxes, just use the
JSON syntax, it should be expressive enough.

On the other hand, doing things like we do in QCOw2 is also a valid
argument.

>> address@hidden: The format string of the file.
>
> Nice that format is mandatory.  Do we want to call out a finite list of
> supported formats, or leave it open-ended in this spec?

I'd expect QEMU's implementation of QBM to recognize all formats QEMU
recognizes elsewhere.  Nailing it down here would add yet another place
to update when we change the list of formats QEMU recognizes.  I'd
rather not do that.

If you want to get fancy, do something like 'implementations shall
implement format "raw", and may implement additional formats.'

>> +
>> +=== BITMAP schema ===
>> +
>> +A BITMAP dictionary records the information of a bitmap (such as a dirty 
>> bitmap
>> +or a block allocation status bitmap). It has following mandatory fields:
>> +
>> address@hidden: The name of the bitmap file. The bitmap file is in little 
>> endian, both
>
> s/in //
>
>> +       byte-order-wise and bit-order-wise, which means the LSB in the byte 0
>> +       corresponds to the first sectors.
>
> s/the byte 0/the first byte/
>
> Again, should we be reusing something from qapi/block-core.json, to
> allow network devices with more structure than just 'json:...' naming?
>
>> address@hidden: How many bytes of data does one bit in the bitmap track.
>> +                    This value must be a power of 2 and no less than 512.
>> address@hidden: The type of the bitmap.  Currently only "dirty" and 
>> "allocation" are
>> +       supported.
>> +       "dirty" indicates a block dirty bitmap; "allocation" indicates a
>> +       allocation status bitmap. There must be at most one "allocation" 
>> bitmap.
>> +
>> +If the type of the bitmap is "allocation", an extra field "backing" is also
>> +accepted:
>> +
>> address@hidden: a dictionary as specified in the IMAGE schema. It can be 
>> used to
>> +          adding a backing file to raw image.
>
> s/adding/add/
>
> As promised above, would an alternative representation be any better?
> That is, I'm trying to see if I could write qapi to describe the
> structure you've presented here, and I fell short when it comes to
> naming a particular bitmap.  Also, since there is at most one
> 'type':'allocation' member of 'bitmaps', I wonder if separating it out
> would make it easier to locate.
>
> Here's my proposal for an alternative schema, written in qapi:
>
> { 'struct': 'Other', 'data': { ...however we describe a network file... } }
> { 'alternate': 'File', 'data': { 'file': 'str', 'struct': 'Other' } }
> { 'struct': 'Bitmap', 'data': {
>    'name':'str', 'file': 'File',
>    'granularity-bytes':'int' } }
> { 'struct': 'AllocationBitmap', 'base': 'Bitmap', 'data': {
>   '*backing': 'Image' } }
> { 'struct': 'Image', 'data': {
>   'file': 'File',
>   'format': 'str' # would an enum be better?
> } }
> { 'struct': 'Desc', 'data': {
>   'version': 'int', 'image': 'Image',
>   '*allocation': 'AllocationBitmap',
>   'dirty': [ 'AllocationBitmap' ]
> } }
> { 'struct': 'QBM', 'data': { 'QBM': 'Desc' } }
>
> where the json description file must consist of a single 'QBM' qapi
> struct, and where the use of a QAPI alternate type 'File' allows us to
> specify either a file name or a formal structure for describing a
> network resource.  Below, I'll rewrite your example with my schema...
>
>> +
>> +
>> +=== Extended fields ===
>> +
>> +Implementations are allowed to extend the format schema by inserting 
>> additinoal
>
> s/additinoal/additional/
>
>> +members into above dictionaries, with key names that starts with either
>> +an "ext-hard-" or an "ext-soft-" prefix.

Elsewhere in QEMU, we use "x-" prefixes.

> Should we be more like qcow2 and have a third category of auto-clear
> keys (if you don't recognize the key, remove it upon editing the file,
> but reading is okay)?  Feature negotiation via this approach requires
> reading every member of 'bitmaps' (well, we have to do that anyway to
> parse the full JSON structure); would it be any better to have an
> up-front section in the top level that describes what features are in
> use, rather than requiring all new features to use the 'ext-' namespace?
>
> Should we require hard failure on any key whose name is not recognized
> (other than the weirdness of your proposal having the keys of 'bitmaps'
> be user-supplied names)?
>
>> +
>> +Extended fields prefixed with "ext-soft-" are optional and can be ignored by
>> +parsers if they do not support it; fields starting with "ext-hard-" are
>> +mandatory and cannot be ignored, a parser should not proceed parsing the 
>> image
>> +if it does not support it.
>
> Is it really the entire image invalidated if an extension is tied to a
> particular bitmap, or only that bitmap?
>
>> +
>> +It is strongly recommended that the application names are also included in 
>> the
>> +extention name string, such as "ext-hard-qemu-", if the effect or
>
> s/extention/extension/
>
>> +interpretation of the field is local to a specific application.
>> +
>> +For example, QEMU can implement a "checksum" feature to make sure no files
>> +referred to by the json descriptor are modified inconsistently, by adding
>> +"ext-soft-qemu-checksum" fields in "image" and "bitmaps" descriptions, like 
>> in
>> +the json text found below.
>
> If an extension proves to be useful, how do we standardize it later?
> Will it always have to carry the 'ext-' prefix?

I'd say no.  By definition above, "ext-" is for implementation's
extensions.  Using it for standard features as well would be bad taste
and confusing.

> You said that soft extensions can be ignored on parse - but if we write
> to the file, couldn't we possibly be invalidating the contents of the
> extension field, and not leaving a breadcrumb for the future reader that
> understands the extension to know that we messed it up?  I think an
> auto-clear feature would be useful (preserve the checksum field, but
> clear the associated auto-clear so that the newer reader knows that the
> checksum has to be checked).
>
> Should we be thinking about a write lock extension (similar to the
> current thread on qcow2 write locks), to make it less likely that two
> writers will be modifying the descriptor, image file, and/or bitmaps at
> the same time?
>
>> +
>> +=== QBM descriptor file example ===
>> +
>> +This is the content of a QBM image's json descriptor file, which contains a
>> +data image (data.img), and three bitmaps, out of which the "allocation" 
>> bitmap
>> +associates a backing file to this image (base.img).
>> +
>> +{ "QBM": {
>> +    "version": 1,
>> +    "image": {
>> +        "file": "data.img",
>> +        "format": "raw"
>> +        "ext-soft-qemu-checksum": "9eff24b72bd693cc8aa3e887141b96f8",
>
> Comma on wrong row.
>
>> +    },
>> +    "bitmaps": {
>> +        "0": {
>> +            "file": "bitmap0.bin",
>> +            "granularity-bytes": 512,
>> +            "type": "dirty"
>> +        },
>> +        "1": {
>> +            "file": "bitmap1.bin",
>> +            "granularity-bytes": 4096,
>> +            "type": "dirty"
>> +        },
>> +        "2": {
>> +            "file": "bitmap3.bin",
>> +            "granularity-bytes": 4096,
>> +            "type": "allocation"
>
> Missing comma
>
>> +            "backing": {
>> +                "file": "base.img",
>> +                "format": "raw"
>> +                "ext-soft-qemu-checksum": 
>> "fcad1f672b2fb19948405e7a1a18c2a7",
>
> Comma on wrong row
>
>> +            },
>
> No trailing commas in JSON
>
> Your approach means that the same bitmap cannot be used for both 'dirty'
> and 'allocation' (because all entries in the 'bitmaps' dictionary must
> have distinct names).  (Although nothing stops two bitmap entries from
> naming the same 'file'.)
>
>> +        }
>> +    }
>> +} }
>
> So, rewriting to my schema above, this would be:
>
> { "QBM": {
>     "version": 1,
>     "image": {
>         "file": "data.img",
>         "format": "raw",
>         "ext-soft-qemu-checksum": "9eff24b72bd693cc8aa3e887141b96f8"
>     },
>     "allocation": {
>         "name": "2",
>         "file": "bitmap3.bin",
>         "granularity-bytes": 4096,
>         "backing": {
>             "file": "base.img",
>             "format": "raw"
>             "ext-soft-qemu-checksum": "fcad1f672b2fb19948405e7a1a18c2a7"
>         }
>     },
>     "bitmaps": [
>         {
>             "name": "0",
>             "file": "bitmap0.bin",
>             "granularity-bytes": 512
>         },
>         {
>             "name": "1",
>             "file": "bitmap1.bin",
>             "granularity-bytes": 4096
>         }
>     ]
> } }
>
>> +
>> 
>
> Awkward to end in a blank line.