Re: [Qemu-block] [Qemu-devel] [RFC PATCH 01/16] doc: Add QBM format specification

[John Snow]

On 01/26/2016 12:51 PM, Eric Blake wrote:
> On 01/26/2016 03:38 AM, Fam Zheng wrote:
>> Signed-off-by: Fam Zheng <address@hidden>
>> ---
>>  docs/specs/qbm.md | 118 
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>  1 file changed, 118 insertions(+)
>>  create mode 100644 docs/specs/qbm.md
>>
>> diff --git a/docs/specs/qbm.md b/docs/specs/qbm.md
>> new file mode 100644
>> index 0000000..b91910b
>> --- /dev/null
>> +++ b/docs/specs/qbm.md
>> @@ -0,0 +1,118 @@
>> +QEMU Block Bitmap (QBM)
>> +=======================
> 
> No explicit copyright mention means that this document is GPLv2+ by
> default.  I don't know if any 3rd-party implementation trying to use
> this spec would object to that, or if a looser license is desirable here
> (I don't personally care, but just raising the point).
> 
>> +
>> +QBM is a multi-file disk format to allow storing persistent block bitmaps 
>> along
>> +with the tracked data image.  A QBM image includes one json descriptor file,
>> +one data image, one or more bitmap files that describe the block dirty 
>> status
> 
> s/one or more/and one or more/
> 
> Must it have block dirty status bitmaps, or can you have a QBM image
> with just an allocation bitmap?
> 

Intent seems to be one or more of any kind.

>> +of the data image.
>> +
>> +The json file describes the structure of the image. The structure of the 
>> json
>> +descriptor file is:
> 
> Please mention that the file must be valid JSON per RFC 7159.  Probably
> also worth requiring that the file is a text file (ends in a newline).
> 
>> +
>> +    QBM-JSON-FILE := { "QBM": DESC-JSON }
>> +
>> +    DESC-JSON := { "version": 1,
>> +                   "image": IMAGE,
>> +                   "BITMAPS": BITMAPS
> 
> s/"BITMAPS"/"bitmaps"/
> 
> See my thoughts below on whether this is the ideal top-level structure.
> 
>> +                 }
>> +
>> +Fields in the top level json dictionary are:
>> +
>> address@hidden: An integer which must be 1.
>> address@hidden: A dictionary in IMAGE schema, as described later. It 
>> provides the
>> +        information of the data image where user data is stored. Its format 
>> is
>> +        documented in the "IMAGE schema" section.
>> address@hidden: A dictionary that describes one ore more bitmap files. The 
>> keys into
>> +          the dictionary are the names of bitmap, which must be strings, and
>> +          each value is a dictionary describing the information of the 
>> bitmap,
>> +          as documented below in the "BITMAP schema" section.
> 
> Making 'bitmaps' be a dictionary means that we now have keys that might
> not be an identifier.  Although this is valid JSON, it may trip up some
> tools.  Would it be better to make 'bitmaps' be a list of dictionaries,
> where each dictionary has a 'name':'value' member, so that bitmap names
> that are not a (C, Python, whatever) identifier are still valid?
> 
>> +
>> +=== IMAGE schema ===
>> +
>> +An IMAGE records the information of an image (such as a data image or a 
>> backing
>> +file). It has following fields:
> 
> I liked how you showed DESC-JSON := for the top level; should you do the
> same here for IMAGE?
> 
>> +
>> address@hidden: The file name string of the referenced image. If it's a 
>> relative path,
>> +       the file should be found relative to the descriptor file's
>> +       location.
> 
> Does that mean we'll have to use 'json:...' encoding for representing
> network resources?  Should we instead reuse some of the
> qapi/block-core.json representation of a block device?
> 

This is an interesting thought: If the design of QBM is such that it is
externally useful to other VM storage utilities, it might be best to
avoid baking in QEMU-specific 'isms for the storage specifier.

Of course, that still leaves us up a creek without a paddle to describe
more complicated image formats, so perhaps we should spell out that the
"filename" can be any kind of identifier (including more json) that
describes the resource. For QEMU, this means our block device json format.

Other utilities will have to be prepared for the notion that this might
not be a straight filename, and they may not know how to interpret it.

Is that acceptable?

>> address@hidden: The format string of the file.
> 
> Nice that format is mandatory.  Do we want to call out a finite list of
> supported formats, or leave it open-ended in this spec?
> 

Right, QEMU's usage of "qcow2" "raw" etc are not necessarily following
some global standard (unless -- are we?) and we should either spell out
the expected values or point to some namespace.

>> +
>> +=== BITMAP schema ===
>> +
>> +A BITMAP dictionary records the information of a bitmap (such as a dirty 
>> bitmap
>> +or a block allocation status bitmap). It has following mandatory fields:
>> +
>> address@hidden: The name of the bitmap file. The bitmap file is in little 
>> endian, both
> 
> s/in //
> 
>> +       byte-order-wise and bit-order-wise, which means the LSB in the byte 0
>> +       corresponds to the first sectors.
> 
> s/the byte 0/the first byte/
> 
> Again, should we be reusing something from qapi/block-core.json, to
> allow network devices with more structure than just 'json:...' naming?
> 
>> address@hidden: How many bytes of data does one bit in the bitmap track.
>> +                    This value must be a power of 2 and no less than 512.
>> address@hidden: The type of the bitmap.  Currently only "dirty" and 
>> "allocation" are
>> +       supported.
>> +       "dirty" indicates a block dirty bitmap; "allocation" indicates a
>> +       allocation status bitmap. There must be at most one "allocation" 
>> bitmap.
>> +
>> +If the type of the bitmap is "allocation", an extra field "backing" is also
>> +accepted:
>> +
>> address@hidden: a dictionary as specified in the IMAGE schema. It can be 
>> used to
>> +          adding a backing file to raw image.
> 
> s/adding/add/
> 
> As promised above, would an alternative representation be any better?
> That is, I'm trying to see if I could write qapi to describe the
> structure you've presented here, and I fell short when it comes to
> naming a particular bitmap.  Also, since there is at most one
> 'type':'allocation' member of 'bitmaps', I wonder if separating it out
> would make it easier to locate.
> 
> Here's my proposal for an alternative schema, written in qapi:
> 

Hesitant to use QEMUisms for describing data in a spec.

> { 'struct': 'Other', 'data': { ...however we describe a network file... } }
> { 'alternate': 'File', 'data': { 'file': 'str', 'struct': 'Other' } }
> { 'struct': 'Bitmap', 'data': {
>    'name':'str', 'file': 'File',
>    'granularity-bytes':'int' } }
> { 'struct': 'AllocationBitmap', 'base': 'Bitmap', 'data': {
>   '*backing': 'Image' } }
> { 'struct': 'Image', 'data': {
>   'file': 'File',
>   'format': 'str' # would an enum be better?
> } }
> { 'struct': 'Desc', 'data': {
>   'version': 'int', 'image': 'Image',
>   '*allocation': 'AllocationBitmap',
>   'dirty': [ 'AllocationBitmap' ]
> } }
> { 'struct': 'QBM', 'data': { 'QBM': 'Desc' } }
> 
> where the json description file must consist of a single 'QBM' qapi
> struct, and where the use of a QAPI alternate type 'File' allows us to
> specify either a file name or a formal structure for describing a
> network resource.  Below, I'll rewrite your example with my schema...
> 
>> +
>> +
>> +=== Extended fields ===
>> +
>> +Implementations are allowed to extend the format schema by inserting 
>> additinoal
> 
> s/additinoal/additional/
> 
>> +members into above dictionaries, with key names that starts with either
>> +an "ext-hard-" or an "ext-soft-" prefix.
> 
> Should we be more like qcow2 and have a third category of auto-clear
> keys (if you don't recognize the key, remove it upon editing the file,
> but reading is okay)?  Feature negotiation via this approach requires
> reading every member of 'bitmaps' (well, we have to do that anyway to
> parse the full JSON structure); would it be any better to have an
> up-front section in the top level that describes what features are in
> use, rather than requiring all new features to use the 'ext-' namespace?
> 
> Should we require hard failure on any key whose name is not recognized
> (other than the weirdness of your proposal having the keys of 'bitmaps'
> be user-supplied names)?
> 

Sounds appropriate, if we use name:value pairs.

>> +
>> +Extended fields prefixed with "ext-soft-" are optional and can be ignored by
>> +parsers if they do not support it; fields starting with "ext-hard-" are
>> +mandatory and cannot be ignored, a parser should not proceed parsing the 
>> image
>> +if it does not support it.
> 
> Is it really the entire image invalidated if an extension is tied to a
> particular bitmap, or only that bitmap?
> 

Going to go ahead and say "yes." If we tie allocation bitmaps to images,
it's a bit rude to reach in and fiddle with the image if just this QBM
descriptor is valid.

>> +
>> +It is strongly recommended that the application names are also included in 
>> the
>> +extention name string, such as "ext-hard-qemu-", if the effect or
> 
> s/extention/extension/
> 
>> +interpretation of the field is local to a specific application.
>> +
>> +For example, QEMU can implement a "checksum" feature to make sure no files
>> +referred to by the json descriptor are modified inconsistently, by adding
>> +"ext-soft-qemu-checksum" fields in "image" and "bitmaps" descriptions, like 
>> in
>> +the json text found below.
> 
> If an extension proves to be useful, how do we standardize it later?
> Will it always have to carry the 'ext-' prefix?
> 

Interesting question. Perhaps future revisions of the spec will have to
specify aliases for ext-** properties that get canonicalized.

> You said that soft extensions can be ignored on parse - but if we write
> to the file, couldn't we possibly be invalidating the contents of the
> extension field, and not leaving a breadcrumb for the future reader that
> understands the extension to know that we messed it up?  I think an
> auto-clear feature would be useful (preserve the checksum field, but
> clear the associated auto-clear so that the newer reader knows that the
> checksum has to be checked).
> 

Shame on the tool that used "soft" for a hard requirement, then. Point
taken, though, the checksum would be a prime candidate for an autoclear
field.

Or, at the very least, adding a breadcrumb field that's essentially "A
tool modified this without understanding all of the soft fields" sounds
nice.

> Should we be thinking about a write lock extension (similar to the
> current thread on qcow2 write locks), to make it less likely that two
> writers will be modifying the descriptor, image file, and/or bitmaps at
> the same time?
> 

Sounds "nice to have" but not necessary for V1. I think it's well
understood that opening a file in two places will lead to heartbreak and
ruin.

>> +
>> +=== QBM descriptor file example ===
>> +
>> +This is the content of a QBM image's json descriptor file, which contains a
>> +data image (data.img), and three bitmaps, out of which the "allocation" 
>> bitmap
>> +associates a backing file to this image (base.img).
>> +
>> +{ "QBM": {
>> +    "version": 1,
>> +    "image": {
>> +        "file": "data.img",
>> +        "format": "raw"
>> +        "ext-soft-qemu-checksum": "9eff24b72bd693cc8aa3e887141b96f8",
> 
> Comma on wrong row.
> 
>> +    },
>> +    "bitmaps": {
>> +        "0": {
>> +            "file": "bitmap0.bin",
>> +            "granularity-bytes": 512,
>> +            "type": "dirty"
>> +        },
>> +        "1": {
>> +            "file": "bitmap1.bin",
>> +            "granularity-bytes": 4096,
>> +            "type": "dirty"
>> +        },
>> +        "2": {
>> +            "file": "bitmap3.bin",
>> +            "granularity-bytes": 4096,
>> +            "type": "allocation"
> 
> Missing comma
> 
>> +            "backing": {
>> +                "file": "base.img",
>> +                "format": "raw"
>> +                "ext-soft-qemu-checksum": 
>> "fcad1f672b2fb19948405e7a1a18c2a7",
> 
> Comma on wrong row
> 
>> +            },
> 
> No trailing commas in JSON
> 
> Your approach means that the same bitmap cannot be used for both 'dirty'
> and 'allocation' (because all entries in the 'bitmaps' dictionary must
> have distinct names).  (Although nothing stops two bitmap entries from
> naming the same 'file'.)
> 

I think that's fine, how often will the allocation bitmap align with the
dirty tracking bitmap? It will at first, but it's likely to diverge
fairly often.

>> +        }
>> +    }
>> +} }
> 
> So, rewriting to my schema above, this would be:
> 
> { "QBM": {
>     "version": 1,
>     "image": {
>         "file": "data.img",
>         "format": "raw",
>         "ext-soft-qemu-checksum": "9eff24b72bd693cc8aa3e887141b96f8"
>     },
>     "allocation": {
>         "name": "2",
>         "file": "bitmap3.bin",
>         "granularity-bytes": 4096,
>         "backing": {
>             "file": "base.img",
>             "format": "raw"
>             "ext-soft-qemu-checksum": "fcad1f672b2fb19948405e7a1a18c2a7"
>         }
>     },
>     "bitmaps": [
>         {
>             "name": "0",
>             "file": "bitmap0.bin",
>             "granularity-bytes": 512
>         },
>         {
>             "name": "1",
>             "file": "bitmap1.bin",
>             "granularity-bytes": 4096
>         }
>     ]
> } }
> 
>> +
>>
> 
> Awkward to end in a blank line.
>