Re: [Qemu-devel] blobstore disk format (was Re: Design of the blobstore)

[Michael S. Tsirkin]

On Mon, Sep 19, 2011 at 12:22:02PM -0400, Stefan Berger wrote:
> On 09/17/2011 03:28 PM, Michael S. Tsirkin wrote:
> >On Fri, Sep 16, 2011 at 12:46:40PM -0400, Stefan Berger wrote:
> >>On 09/16/2011 10:44 AM, Michael S. Tsirkin wrote:
> >>>On Thu, Sep 15, 2011 at 10:33:13AM -0400, Stefan Berger wrote:
> >>>>On 09/15/2011 08:28 AM, Michael S. Tsirkin wrote:
> >>>>>So the below is a proposal for a directory scheme
> >>>>>for storing (optionally multiple) nvram images,
> >>>>>along with any metadata.
> >>>>>Data is encoded using BER:
> >>>>>http://en.wikipedia.org/wiki/Basic_Encoding_Rules
> >>>>>Specifically, we mostly use the subsets.
> >>>>>
> >>>>Would it change anything if we were to think of the NVRAM image as
> >>>>another piece of metadata?
> >>>Yes, we can do that, sure. I had the feeling that it will help to lay
> >>>out the image at the end, to make directory listing
> >>>more efficient - the rest of metadata is usually small,
> >>>image might be somewhat large.
> >>>
> >>Why not let a convenience library handle the metadata on the device
> >>level, having it create the blob that the NVRAM layer ends up
> >>writing and parsing before the device uses it? Otherwise I should
> >>maybe rename the nvram to meatdata_store :-/
> >Maybe we are talking about different things. All I agrue for
> >is using a common standard format for storing metadata,
> >instead of having each device roll its own.
> That's fine. The TPM code inside libtpms serializes all internal
> data structures for later resumption. It doesn't use ASN.1 but in
> effect endianess-normalizes them and stores them in a packed format
> that can later be resumed along with a version tag prepended where
> need . Are you suggesting to change that ? I hope not...

I'm guessing that it makes sense to use whatever libtpms does
as is - if we change the format we'd have to parse it ourselves.
We can use it as a binary blob.
It's only when we add our own versioning/structure
on top of it, such as adding actual size, where
using a standard format makes sense IMO.

> >>>>I am also wondering whether each device shouldn't just handle the
> >>>>metadata itself,
> >>>It could be that just means we will have custom code with
> >>>different bugs in each device.
> >>>Note that from experience with formats, the problem with
> >>>time becomes less trivial than it seems as we
> >>>need to provide forward and backward compatibility
> >>>guarantees.
> >>>
> >>Is that guaranteed just by using ASN.1 ?
> >At least for BER, yes. We can always skip an optional field
> >that we don't recognize without knowing anything about
> >its internal format.
> 
> >>Do we need to add a
> >>revision to the metadata?
> >IMO, no. Instead we add optional attributes as long as we can
> >preserve backwards compatibility, and madatory attributes
> >if we can't.
> >
> Are devices doing this right now or are these future changes to
> devices' code?

The don't do right now.
AFAIK Anthony's working on changing devices to use a visitor
for migration, and I want to add a visitor backend
that will use BER format.

> >>>>encryption, integrity value (crc32 or sha1) and so on. What
> >>>>metadata should there be that really need to be handled on the NVRAM
> >>>>API and below level rather than on the device-specific code level?
> >>>So checksum  (checksum value and type) 'and so on' are what I call
> >>>metadata :) Doing it at device level seems wrong.
> >>>
> >>You mean doing it at the NVRAM level seems wrong. Of course, again
> >>something a device could write into a header prepended to the actual
> >>blob. Maybe every device that needs it should do that so that if we
> >>were to support encryption of blobs and the key for decryption was
> >>wrong one could detect it early without feeding badly decrypted /
> >>corrupted state into the device and see what happens.
> >Do what? Checksum the data? Well, error detection is nice,
> >but it could be that people actually care about not losing
> >all of the data on nvram if qemu is killed.  I also wonder whether
> >invalidating all data because of a single bit corruption is a bug or a
> >feature.
> >
> The checksuming I think makes sense if encryption is being added so
> decryption and testing for proper key material remains an NVRAM
> operation rather than a device operation.

Not sure how this addresses the question of what to do
on checksum failure.

> >>>>>We use a directory as a SET in a CER format.
> >>>>>This allows generating directory online without scanning
> >>>>>the entries beforehand.
> >>>>>
> >>>>I guess it is the 'unknown' for me... but what is the advantage of
> >>>>using ASN1 for this rather than just writing out packed and
> >>>>endianess-normalized data structures (with revision value),
> >>>If you want an example of where this 'custom formats are easy
> >>>so let us write one' leads to in the end,
> >>>look no further than live migration code.
> >>>It's a mess of hacks that does not even work across
> >>>upstream qemu versions, leave alone across
> >>>downstreams (different linux distros).
> >>>
> >>So is ASN1 the answer or does one still need to add a revision tag
> >>to each blob putting in custom code for parsing the different
> >>revisions of data structures (I guess) that may be extended/changed
> >>over time?
> >>
> >>    Stefan
> >We don't need revisions. We can always parse a new structure
> >skipping optional attributes we don't recognize. In case we want to
> >break old qemu versions intentially, we can add
> >a mandatory attribute.
> So you said you had some code for the handling of ASN.1. Can sketch
> how the interaction of devices would work with mandatory and
> optional attributes along with an API?

So for now what I'm happy with is a low level BER parser API.
NVRAM would implement a scheme on top of that
and bdrv. I'll pack up what I have and post ASAP.

> I'd prefer to NOT have the
> attributes and values be a part of the NVRAM API itself but let a
> (mandatory) library handle the serialization and deserialization of
> these metadata when a device wants to write or read state
> respectively. But maybe I just want to keep the NVRAM API 'too
> simple'.
> 
>    Stefan

I'm fine with that. Still tpm has some device specific
things like 'actual size' so the api will need to address that
in some generic way.

> >>>>having
> >>>>them crc32-protected to have some sanity checking in place?
> >>>>
> >>>>     Stefan
> >>>I'm not sure why we want crc specifically in TPM.
> >>>If it is 'just because we can' then it probably
> >>>applies to other non-volatile storage?
> >>>Storage generally?
> >>>
> >>>>>The rest of the encoding uses a DER format.
> >>>>>This makes for fast parsing as entries are easy to skip.
> >>>>>
> >>>>>Each entry is encoded in DER format.
> >>>>>Each entry is a SEQUENCE with two objects:
> >>>>>1. nvram
> >>>>>2. optional name - a UTF8String
> >>>>>
> >>>>>Binary data is stored as OCTET-STRING values on disk.
> >>>>>Any RW metadata is stored as OCTET-STRING value as well.
> >>>>>Any RO metadata is stored in appropriate universal encoding,
> >>>>>by type.
> >>>>>
> >>>>>On the context below, an attribute is either a IA5String or a SEQUENCE.
> >>>>>If IA5String, this is the attribute name, and it has no value.
> >>>>>If SEQUENCE, the first entry in the sequence is an
> >>>>>IA5String, it is the attribute name. The rest of the entries
> >>>>>represent the attribute value.
> >>>>>
> >>>>>Mandatory/optional attributes: depends on type.
> >>>>>tpm will have realsize as RW mandatory attribute.
> >>>>>
> >>>>>Each nvram is built as a SEQUENCE including 4 objects
> >>>>>1. type - an IA5String. downstreams can use other types such as
> >>>>>                      UUIDs instead to ensure no conflicts with upstream
> >>>>>2. SET of mandatory attributes
> >>>>>3. SET of optional attributes
> >>>>>4. data - a RW OCTET-STRING
> >>>>>
> >>>>>It is envisioned that attributes won't be too large,
> >>>>>so they can easily be kept in memory.
> >>>>>
> >>>>>