Re: [Qemu-devel] WIP: Migration format: ASN.1/BER schema

[Dr. David Alan Gilbert]

* Michael S. Tsirkin (address@hidden) wrote:
> On Fri, Mar 07, 2014 at 04:55:03PM +0000, Dr. David Alan Gilbert wrote:

<snip>

> I think this is already useful.
> For example this will make input parser
> more robust against unexpected input
> as octet string blobs are bounded, so if we fail
> to parse one we at least will not
> interpret it as beginning of the next one.

I've also managed (with a slightly improved schema
at the end) to validate my output sequence against the schema
using asn1c (once I'd persuaded it to build stuff and turn
it's debug on).

> I think it's a good idea to merge this gradually,
> even if this means changing format several times:
> as long as we can stay compatible with old machine
> types.

Yes, that would make the merge easier.

> > --------------------------------------------------------------------------------
> > 
> > Qemu {}
> > DEFINITIONS IMPLICIT TAGS ::=
> > BEGIN
> > 
> > -- Some basic types used in multiple places --
> > QemuString ::= UTF8String (SIZE (1..255))
> > 
> > -- TODO: 4096 is actually page size whatever that is
> > FullPage ::= OCTET STRING (SIZE (4096))
> > 
> > RAMBlockID ::= SEQUENCE {
> >   name   QemuString,
> >   len    INTEGER
> 
> is len needed here?
> 
> > }
> > 
> > RAMSecEntry ::= [ APPLICATION 8914 ] SEQUENCE {
> >   addr         INTEGER,   -- Address or offset or size
> 
> hmm so which one?
> 
> >   flags        INTEGER,   -- maybe more explicit type?
> >   name         QemuString OPTIONAL,
> > 
> >   body         CHOICE {
> >     bl         SEQUENCE OF RAMBlockID,
> >     compr      INTEGER (0..255), -- Page filled with this value
> >     page       FullPage
> >     -- TODO xbzrle --
> 
> this one confused me.
> I expected RAMBlockID followed by a sequence of pages

I've kept with the structure of the way the migrate works at the moment;
it transmits the 'shape' of RAM as a list of name/length pairs, and only
once it's validated that it has all the RAM blocks and their lengths match
does it start shifting pages across.

This a RAM section is either the initialisation data (the sequence of 
RAMBlockID's)
or a stream of pages.
I don't think it's a bad idea to have that RAM list at the beginning,
it catches a lot of silly mismatches - if you've ever hit a complaint
about a mismatched BIOS length or a missing ROM, it's probably come
from the check of this incoming list.

The existing binary format is a stream of things separated by a
64bit int that is an address/offset/or size depending on some bits OR'd
into the bottom of the address; I'd currently split the flags out
of that 64bit int, but hadn't split that further yet.
I'm thinking of splitting this down further, especially so I can
reduce the size of 'all 0' pages.

> > 
> > RAMSecList ::= [ APPLICATION 9810 ] SEQUENCE OF RAMSecEntry
> > 
> > SubSection ::= [ APPLICATION 10707 ] SEQUENCE {
> >   name         QemuString,
> >   versionid    INTEGER,
> > 
> >   contents     SEQUENCE OF VMStateEntries
> > }
> > 
> > SubSecList ::= [ APPLICATION 10700 ] SEQUENCE OF SubSection
> > 
> > VMStateEntries ::= CHOICE {
> >   -- Hmm need to think more --
> 
> I'm guessing individual devices will have
> tagged entries here?

I'm thinking of *allowing* individual devices to have
tagged entries here, but I don't think I'd require it  - since
to require it would mean I'd have to change every existing
VMState to give it a tag.
So I was going to allow VMStateEntries to be a generic set
using universal tags, which we could add specific entries
where we'd given common devices full tags.

> >   dummy    INTEGER,
> 
> why is this here?

It's one of the universal entries; my current full version (that
verifies) is:

VMStateEntries ::= CHOICE {
  -- Hmm need to think more --
  array       SEQUENCE OF VMStateEntries,
  bool        BOOLEAN,
  int         INTEGER,
  oldblob     OCTET STRING,
  subsecl     SubSecList
}

> 
> >   subsecl  SubSecList,
> >   oldblob  OCTET STRING
> > }
> > 
> > VMState ::= SEQUENCE OF VMStateEntries
> > 
> > -- Restrict to unsigned?
> > SectionID ::= INTEGER
> > 
> > SecFull ::= [ APPLICATION 2003 ] SEQUENCE {
> >   name         QemuString,
> >   sectionid    SectionID,
> 
> what is this id exactly?

It's again a hold over from the old format; Sections
come in two varieties 'Full' and 'Min'; 'Full'
sections are typically used for devices that just get
sent once, and thus have a name - the ID doesn't
really do much for them; however devices that send
data iteratively send a Full followed by a series of
Min's - each Min just has the SectionID matching the Full,
so it saves it having to send the name each time,
and it also means that it just has an index rather
than having to string compare the name for every
section.

Note that iterative devices can be interleaved so
you might have some RAM, followed by some disk blocks
and then some more RAM etc, which is why you can't
just have a sequence of pages.

Dave
P.S. Here's the current verison of my schema which
validates against my ber data.

Qemu {}
DEFINITIONS IMPLICIT TAGS ::=
BEGIN

-- Some basic types used in multiple places --
QemuString ::= UTF8String (SIZE (1..255))

-- TODO: 4096 is actually page size whatever that is
FullPage ::= OCTET STRING (SIZE (4096))

RAMBlockID ::= SEQUENCE {
  name   QemuString,
  len    INTEGER
}

RAMSecEntry ::= [ APPLICATION 8914 ] SEQUENCE {
  addr         INTEGER,   -- Address or offset or size
  flags        INTEGER,   -- maybe more explicit type?
  name         QemuString OPTIONAL,

  body         CHOICE {
    bl         SEQUENCE OF RAMBlockID,
    compr      INTEGER (0..255), -- Page filled with this value
    page       FullPage
    -- TODO xbzrle --
  }
}

RAMSecList ::= [ APPLICATION 9810 ] SEQUENCE OF RAMSecEntry

SubSection ::= [ APPLICATION 10707 ] SEQUENCE {
  name         QemuString,
  versionid    INTEGER,

  contents     SEQUENCE OF VMStateEntries
}

SubSecList ::= [ APPLICATION 10700 ] SEQUENCE OF SubSection

VMStateEntries ::= CHOICE {
  -- Hmm need to think more --
  array       SEQUENCE OF VMStateEntries,
  bool        BOOLEAN,
  int         INTEGER,
  oldblob     OCTET STRING,
  subsecl     SubSecList
}

VMState ::= SEQUENCE OF VMStateEntries

-- Restrict to unsigned?
SectionID ::= INTEGER

SecFull ::= [ APPLICATION 2003 ] SEQUENCE {
  name         QemuString,
  sectionid    SectionID,
  instanceid   INTEGER,
  versionid    INTEGER,

  contents     CHOICE {
    ramsec     RAMSecList,
    -- TODO other iterator initial stuff --
    vmstate    VMState,
    oldblob  OCTET STRING
  }
}

SecMin ::= [ APPLICATION 211 ] SEQUENCE {
  sectionid   SectionID,

  contents     CHOICE {
    ramsec     RAMSecList
    -- TODO other iterator general/end stuff --
  }
}

Sections ::= CHOICE {
  full    SecFull,
  min     SecMin
}

-- The whole file --
-- Application tag used to get first 32bits of file
-- to come out as 7f cd c5 51  - the 51 is Q
-- the c5 and cd being E,M but with the top bit set
-- which BER requires
QemuFile ::= [ APPLICATION 1270481 ] SEQUENCE {
  version INTEGER (3),
  top     SEQUENCE OF Sections
}

END
--
Dr. David Alan Gilbert / address@hidden / Manchester, UK