Re: [Qemu-devel] [RFC PATCH 01/16] doc: Add QBM format specification

[John Snow]

On 01/26/2016 05:38 AM, Fam Zheng wrote:
> Signed-off-by: Fam Zheng <address@hidden>
> ---
>  docs/specs/qbm.md | 118 
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 118 insertions(+)
>  create mode 100644 docs/specs/qbm.md
> 
> diff --git a/docs/specs/qbm.md b/docs/specs/qbm.md
> new file mode 100644
> index 0000000..b91910b
> --- /dev/null
> +++ b/docs/specs/qbm.md
> @@ -0,0 +1,118 @@
> +QEMU Block Bitmap (QBM)
> +=======================
> +
> +QBM is a multi-file disk format to allow storing persistent block bitmaps 
> along
> +with the tracked data image.  A QBM image includes one json descriptor file,
> +one data image, one or more bitmap files that describe the block dirty status
> +of the data image.
> +
> +The json file describes the structure of the image. The structure of the json
> +descriptor file is:
> +
> +    QBM-JSON-FILE := { "QBM": DESC-JSON }
> +
> +    DESC-JSON := { "version": 1,
> +                   "image": IMAGE,
> +                   "BITMAPS": BITMAPS
> +                 }
> +
> +Fields in the top level json dictionary are:
> +
> address@hidden: An integer which must be 1.
> address@hidden: A dictionary in IMAGE schema, as described later. It provides 
> the
> +        information of the data image where user data is stored. Its format 
> is
> +        documented in the "IMAGE schema" section.
> address@hidden: A dictionary that describes one ore more bitmap files. The 
> keys into
                                             ^or

> +          the dictionary are the names of bitmap, which must be strings, and
> +          each value is a dictionary describing the information of the 
> bitmap,
> +          as documented below in the "BITMAP schema" section.
> +
> +=== IMAGE schema ===
> +
> +An IMAGE records the information of an image (such as a data image or a 
> backing
> +file). It has following fields:
> +
> address@hidden: The file name string of the referenced image. If it's a 
> relative path,
> +       the file should be found relative to the descriptor file's
> +       location.
> address@hidden: The format string of the file.

Are these codified in any spec, or do they only exist as QEMU API? Is it
important to document what namespace these format strings belong to?

> +
> +=== BITMAP schema ===
> +
> +A BITMAP dictionary records the information of a bitmap (such as a dirty 
> bitmap
> +or a block allocation status bitmap). It has following mandatory fields:
> +
> address@hidden: The name of the bitmap file. The bitmap file is in little 
> endian, both
> +       byte-order-wise and bit-order-wise, which means the LSB in the byte 0
> +       corresponds to the first sectors.
> address@hidden: How many bytes of data does one bit in the bitmap track.
> +                    This value must be a power of 2 and no less than 512.
> address@hidden: The type of the bitmap.  Currently only "dirty" and 
> "allocation" are
> +       supported.
> +       "dirty" indicates a block dirty bitmap; "allocation" indicates a
> +       allocation status bitmap. There must be at most one "allocation" 
> bitmap.

Might be worth syncing with Vladimir to create consistent terminology
for these, e.g. a "Dirty Tracking Bitmap."

> +
> +If the type of the bitmap is "allocation", an extra field "backing" is also
> +accepted:
> +
> address@hidden: a dictionary as specified in the IMAGE schema. It can be used 
> to
> +          adding a backing file to raw image.
> +
> +
> +=== Extended fields ===
> +
> +Implementations are allowed to extend the format schema by inserting 
> additinoal
........................................................................^additional
> +members into above dictionaries, with key names that starts with either
> +an "ext-hard-" or an "ext-soft-" prefix.
> +
> +Extended fields prefixed with "ext-soft-" are optional and can be ignored by
> +parsers if they do not support it; fields starting with "ext-hard-" are
> +mandatory and cannot be ignored, a parser should not proceed parsing the 
> image
> +if it does not support it.
> +
> +It is strongly recommended that the application names are also included in 
> the
> +extention name string, such as "ext-hard-qemu-", if the effect or

extension

> +interpretation of the field is local to a specific application.
> +
> +For example, QEMU can implement a "checksum" feature to make sure no files
> +referred to by the json descriptor are modified inconsistently, by adding
> +"ext-soft-qemu-checksum" fields in "image" and "bitmaps" descriptions, like 
> in
> +the json text found below.
> +
> +=== QBM descriptor file example ===
> +
> +This is the content of a QBM image's json descriptor file, which contains a
> +data image (data.img), and three bitmaps, out of which the "allocation" 
> bitmap
> +associates a backing file to this image (base.img).
> +
> +{ "QBM": {
> +    "version": 1,
> +    "image": {
> +        "file": "data.img",
> +        "format": "raw"
> +        "ext-soft-qemu-checksum": "9eff24b72bd693cc8aa3e887141b96f8",
> +    },
> +    "bitmaps": {
> +        "0": {
> +            "file": "bitmap0.bin",
> +            "granularity-bytes": 512,
> +            "type": "dirty"
> +        },
> +        "1": {
> +            "file": "bitmap1.bin",
> +            "granularity-bytes": 4096,
> +            "type": "dirty"
> +        },
> +        "2": {
> +            "file": "bitmap3.bin",
> +            "granularity-bytes": 4096,
> +            "type": "allocation"
> +            "backing": {
> +                "file": "base.img",
> +                "format": "raw"
> +                "ext-soft-qemu-checksum": "fcad1f672b2fb19948405e7a1a18c2a7",
> +            },
> +        }
> +    }
> +} }
> +
> 

I think I agree with Eric that it might be best to use name:value pairs,
unless there is a strong motivator otherwise -- I seem to recall that
this was a way to prevent duplicate named bitmaps within the JSON schema
itself, which is reasonably compelling.