[Qemu-devel] [PATCH v2 00/10] do savevm/migration save/load via Visitor interface

[Michael Roth]

These patches apply on top of origin/master, and can also be obtained from:
git://repo.or.cz/qemu/mdroth.git migration-visitor-v2

- rebased on 9f60639b848944200c3d33a89233d808de0b5a43. Juan: had a pretty
  hard collision with your recent savevm/vmstate cleanups. I think most of
  the issues were reconciled, but some of your checks such as bailing out
  of vmstate_subsection_load() if qemu_peek_buffer() doesn't return the
  length encoded in the stream (for idstr) and peeking and checking subsection
  tags before progressing through the QEMUFile input buffer were not
  compatible with the visitorized approach. The net effect of failing migration
  if these assertions do not hold should still be intact, however.
- reworked QEMUFile<->Visitor lookup routines to support opaque objects and
  register cleanup routines, mainly to allow cleanup of
  QEMUFileInputVisitor/QEMUFileOutputVisitor/etc via qemu_fclose() without
  being aware of the type of Visitor employed.
- fixed coding style issues as part of moving QEMUFile interfaces to
  qemu-file.c

OVERVIEW

This patch series implements a QEMUFile Visitor class that's intended to
abstract away direct calls to qemu_put_*/qemu_get_* for save/load functions.
Currently this is done by always creating a
QEMUFileInputVisitor/QEMUFileOutputVisitor pair with each call to
qemu_fopen_ops() and maintaining a QEMUFile->I/O Visitor mapping. save/load
functions that are to be converted would them simply use lookup routines to get
a Visitor based on their QEMUFile arguments. Once these save/load functions are
all converted, we can then change the interfaces in bulk and switch over to
passing in the Visitor directly.

This series also converts all of vmstate over to using Visitors, leveraging the
existing vmstate hierachical structure to handle namespacing and structuring of
the fields that are fed to the visitor functions, and introduces trace
statements for qemu_put_*/qemu_get_* and visitor API calls to aid in testing.

Migration compatibility is not affected by these changes, as the
qemu_put_*/qemu_get_* calls have a trivial 1-to-1 mapping with the visit_type_*
calls, and the added field names/structure are not sent out over the wire when
using the QEMUFile visitors, but serve simply as placeholders for future
functionality.

Test scripts/procedure are documented on the wiki page:

http://wiki.qemu.org/Features/Migration/Visitor#Code.2FTesting

PLANS

With these patches in place non-vmstate save/load functions can be converted
over to using Visitors incrementally.

Short term (1.1 timeframe), the goal is to implement a new migration protocol
that is self-describing, such that incompatibilities or migration errors can be
more easilly detected. Currently, a simple change in data types for a
particular device can introduce subtle bugs that won't be detected by the
target, since the target interprets the data according to it's own expectation
of what those data types are. Currently the plan is to achieve this by using
ASN.1 BER in place of QEMUFile via a new BERVisitor.

Also planned is using the visitor interface to aid in things like device
introspection (for both vmstate and non-vmstate devices), likely via the
existing QMPOutputVisitor, as well as debugging/testing migration via a well
structured "PrettyPrintVisitor" type of visitor that also retains ordering,
field names, and type information (unlike with QMP/JSON or ASN.1). I've played
around with doing this indirectly to test the conversions by doing some
post-processing on the visit_* trace statements:

...
PIIX3.PCIDevice.config(256)(1).[255] = (uint8) 0x0
PIIX3.PCIDevice.irq_state(4)(4).[0] = (uint32) 0x0
PIIX3.PCIDevice.irq_state(4)(4).[1] = (uint32) 0x0
PIIX3.PCIDevice.irq_state(4)(4).[2] = (uint32) 0x0
PIIX3.PCIDevice.irq_state(4)(4).[3] = (uint32) 0x0
PIIX3.pci_irq_levels_vmstate(4)(4).[0] = (int32) 0x0
PIIX3.pci_irq_levels_vmstate(4)(4).[1] = (int32) 0x0
PIIX3.pci_irq_levels_vmstate(4)(4).[2] = (int32) 0x0
PIIX3.pci_irq_levels_vmstate(4)(4).[3] = (int32) 0x0
i8259.last_irr = (uint8) 0x1
i8259.irr = (uint8) 0x10
i8259.imr = (uint8) 0xb8
i8259.isr = (uint8) 0x0
i8259.priority_add = (uint8) 0x0
i8259.irq_base = (uint8) 0x8
i8259.read_reg_select = (uint8) 0x0
i8259.poll = (uint8) 0x0
i8259.special_mask = (uint8) 0x0
i8259.init_state = (uint8) 0x0
i8259.auto_eoi = (uint8) 0x0
...
slirp.ip_id = (uint16) 0x2
slirp.bootp.bootp_clients(16)(8).[0].allocated = (uint16) 0x1
slirp.bootp.bootp_clients(16)(8).[0].macaddr(6)(1).[0] = (uint8) 0x1
slirp.bootp.bootp_clients(16)(8).[0].macaddr(6)(1).[1] = (uint8) 0x0
slirp.bootp.bootp_clients(16)(8).[0].macaddr(6)(1).[2] = (uint8) 0x0
slirp.bootp.bootp_clients(16)(8).[0].macaddr(6)(1).[3] = (uint8) 0x0
slirp.bootp.bootp_clients(16)(8).[0].macaddr(6)(1).[4] = (uint8) 0x0
slirp.bootp.bootp_clients(16)(8).[0].macaddr(6)(1).[5] = (uint8) 0x0
slirp.bootp.bootp_clients(16)(8).[1].allocated = (uint16) 0x0
slirp.bootp.bootp_clients(16)(8).[1].macaddr(6)(1).[0] = (uint8) 0x1
slirp.bootp.bootp_clients(16)(8).[1].macaddr(6)(1).[1] = (uint8) 0x0
slirp.bootp.bootp_clients(16)(8).[1].macaddr(6)(1).[2] = (uint8) 0x0
slirp.bootp.bootp_clients(16)(8).[1].macaddr(6)(1).[3] = (uint8) 0x0
slirp.bootp.bootp_clients(16)(8).[1].macaddr(6)(1).[4] = (uint8) 0x0
slirp.bootp.bootp_clients(16)(8).[1].macaddr(6)(1).[5] = (uint8) 0x0
slirp.bootp.bootp_clients(16)(8).[2].allocated = (uint16) 0x0
...

It's not completely clear at this point how the latter will fall into place
with other plans such as QOM that would provide similar capabilities, but since
device serialization will likely be done via automatically-generated visitor
patterns, this conversion should at least serve as an incremental step toward
that transition.

Michael Roth (10):
  qapi: add Visitor interfaces for uint*_t and int*_t
  qapi: add QemuFileOutputVisitor
  qapi: add QemuFileInputVisitor
  savevm: move QEMUFile interfaces into qemu-file.c
  qapi: test cases for QEMUFile input/output visitors
  qemu-file: add QEMUFile<->visitor lookup routines
  trace: qemu_(put|get)_(byte|buffer) events
  trace: add trace statements for visitor interface
  qapi: add trace statements to qapi-visit-core.c
  vmstate: use visitors

 Makefile                        |    2 +-
 Makefile.objs                   |   22 +-
 hw/eeprom93xx.c                 |   12 +-
 hw/fw_cfg.c                     |   12 +-
 hw/hw.h                         |   18 +-
 hw/pci.c                        |   38 +-
 hw/twl92230.c                   |   18 +-
 qapi/qapi-visit-core.c          |  119 +++++-
 qapi/qapi-visit-core.h          |   30 ++
 qapi/qemu-file-input-visitor.c  |  328 ++++++++++++
 qapi/qemu-file-input-visitor.h  |   26 +
 qapi/qemu-file-output-visitor.c |  328 ++++++++++++
 qapi/qemu-file-output-visitor.h |   26 +
 qemu-file.c                     |  688 +++++++++++++++++++++++++
 qemu-timer.h                    |    5 +
 savevm.c                        | 1049 ++++++++++++---------------------------
 target-alpha/machine.c          |   13 +-
 target-i386/machine.c           |   49 ++-
 test-visitor.c                  |  404 +++++++++++++++
 trace-events                    |   43 ++
 20 files changed, 2445 insertions(+), 785 deletions(-)
 create mode 100644 qapi/qemu-file-input-visitor.c
 create mode 100644 qapi/qemu-file-input-visitor.h
 create mode 100644 qapi/qemu-file-output-visitor.c
 create mode 100644 qapi/qemu-file-output-visitor.h
 create mode 100644 qemu-file.c

-- 
1.7.4.1