[Qemu-devel] [RFC v2] Use QEMU IDL for device serialization/introspection

[Michael Roth]

These patches are based on origin/master, and can also be obtained from:

git://github.com/mdroth/qemu.git qidl-rfc2

= CHANGES SINCE RFC V1 =

 - QIDL'd cirrus_vga as a more interesting example
 - add support for QIDL-defined properties
 - pass all files through preprocessor before parsing QIDL annotations to
   gracefully handle #define's/#if's/etc
 - use code-injection/macros to make generated visitor/property code
   available to each compilation unit
 - drop schema generation, instead, expose the QIDL-generated, binary-specific
   schemas via QOM to make it easier to analyze migration compatibility given
   2 versions of QEMU
 - drop VMSTATE code generation, focus on exposing enough state that VMSTATE
   can be eventually be layered on top using qom_path + json_path to access
   all required fields
 - wrap annotations in QIDL(...) to better support arguments and namespacing
 - seperate parser (qidl_parser.py) from code-generation (qidl.py)
 - fix various parsing corner cases:
    - handling of embedded, non-pointer structs and nested struct declarations
    - handling of unsigned/union/int types
    - support expressions in QIDL() parameters and field declarations (to
      be evaluated in the context of the generated visitor function)

= OVERVIEW =

The goal of these patches is to explore how we can leverage an IDL to improve
device serialization/migration.

Patches 1-17 are all infrastructure, and the initial commits for the QIDL
parser/code generator/documentation.

Patches 18+ are our first users: the unit tests, RTC, and cirrus_vga.

After annotating a QOM device with QIDL you can up with something
this (using QOM-FUSE as an example):

    address@hidden:~/w/qom$ ls -l qidl/schemas/
    total 0
    drwxr-xr-x 2 mdroth mdroth 4096 Dec 31  1969 CirrusVGAState
    drwxr-xr-x 2 mdroth mdroth 4096 Dec 31  1969 ISACirrusVGAState
    drwxr-xr-x 2 mdroth mdroth 4096 Dec 31  1969 PCICirrusVGAState
    drwxr-xr-x 2 mdroth mdroth 4096 Dec 31  1969 PCIDevice
    drwxr-xr-x 2 mdroth mdroth 4096 Dec 31  1969 RTCState
    -rw-r--r-- 1 mdroth mdroth 4096 Dec 31  1969 type
    drwxr-xr-x 2 mdroth mdroth 4096 Dec 31  1969 VGACommonState

    address@hidden:~/w/qom$ ls -l machine/unattached/device\[8\]/
    total 0
    -rw-r--r-- 1 mdroth mdroth   4096 Dec 31  1969 addr
    -rw-r--r-- 1 mdroth mdroth   4096 Dec 31  1969 command_serr_enable
    -rw-r--r-- 1 mdroth mdroth   4096 Dec 31  1969 legacy-addr
    -rw-r--r-- 1 mdroth mdroth   4096 Dec 31  1969 legacy-command_serr_enable
    -rw-r--r-- 1 mdroth mdroth   4096 Dec 31  1969 legacy-multifunction
    -rw-r--r-- 1 mdroth mdroth   4096 Dec 31  1969 legacy-romfile
    -rw-r--r-- 1 mdroth mdroth   4096 Dec 31  1969 multifunction
    lrwxr-xr-x 2 mdroth mdroth   4096 Dec 31  1969 parent_bus -> 
../../../machine/i440fx/pci.0
    -rw-r--r-- 1 mdroth mdroth   4096 Dec 31  1969 rombar
    -rw-r--r-- 1 mdroth mdroth   4096 Dec 31  1969 romfile
    -rw-r--r-- 1 mdroth mdroth 175236 Dec 31  1969 state
    lrwxr-xr-x 2 mdroth mdroth   4096 Dec 31  1969 state_schema -> 
../../../qidl/schemas/PCICirrusVGAState
    -rw-r--r-- 1 mdroth mdroth   4096 Dec 31  1969 type

    address@hidden:~/w/qom$ cat machine/unattached/device\[8\]/state
    {u'cirrus_vga': {u'cirrus_blt_srcpitch': 0, u'cirrus_blt_width': 0, \
    u'cirrus_blt_bgcol': 0, u'last_hw_cursor_size': 0, \
    u'real_vram_size': 4194304, u'cirrus_blt_pixelwidth': 0, \
    u'cirrus_blt_mode': 0, u'last_hw_cursor_y_start': 0, \
    u'cirrus_blt_srcaddr': 0, u'cirrus_srccounter': 0, \
    u'cirrus_hidden_palette': [0, 0, 0, 0, 0, ...

At this point I'd like to push to get QIDL in place to expose guest-visible
device state via qom-get so it can be used for pre/post-migration
analysis/verification of device state. I think this is useful in-and-of
itself, and should be fairly low risk.

With the infrastructure in place we can then look at leveraging it for
vmstate and/or a new protocol, and also to figure out a nice way to add
capabilities-based transformations for post-serialize/pre-deserialize to
improve migration compatibility for any wire protocol (QEMUFile/vmstate
or otherwise) we layer on top. VMState at least can be done incrementally, so
we can begin looking at this immediately.

= General/Future Plans =

This is all very much open to discussion, and I only have a general idea of
how we can leverage this to improve migration compatibility/support. That
said:

With everything in place, we'd now have a means to serialize device state into
QObjects (or whatever). We can then perform whatever transformations/mark-up
we need to do (based on capabilities negotation centering around per-device
capabilities, for example), and send the state over the wire for migration.

The wire protocol is then simply a matter of how we encode the transformed
QObject. So a BER protocol could be implemented by creating something analagous
to the JSON-encoding facilities in qjson.c. Or, we could just keep using JSON,
perhaps with compression on top.

Eventually we can extend this approach to send device properties and encode the
actual composition tree in such a way that we can create machine machines on
the target side and avoid the need to duplicate the command-line invocation,
though that will take some substantial re-architecting/removal of the various
factory interfaces and further QOMification.

I'm not planning on incorporating memory migration into this, but it may be
possible to extend this approach to events/data blocks as well.