[Qemu-devel] [PATCH 00/36] post-Eric's fixes, QAPI improvements

[marcandre . lureau]

From: Marc-André Lureau <address@hidden>

Hi,

I have grown a qapi branch during the past 2 months that was
post-poned for review until the introspection and other fixes got
merged or ready.

It could be splitted roughly in 3 parts, but since they depend on each
other, it make sense to send as one:
 1. generate QAPI documentation from schema (1->12)
 2. remove use generated qmp dispatch in monitor (13->20)
 3. add some support for async commands (22->36)

1 and 2 are hopefully not controversial, however 3 was discussed
previously in 'async commands with QMP' thread:

QMP was initially designed with async support, but the implementation
was buggy, since the async context didn't hold the request 'id', and
thus it wasn't really used, so it was removed in 65207c59d a few
months ago.

However, there a valid reasons to want better async support. Many qmp
operations take time, and it's not a good idea to block the monitor
during this time. Also, some operations, such as screendump with QXL,
have to reenter the loop in order to complete. Furthermore, the
current protocol is not purely synchronous, since the client have to
handle unexpected in the middle replies, such as events.

The current status-quo for async commands is to return immediately,
and send loosely related events later. This is quite poor from an API
point of view, and the events are not related to a previous command
(they don't hold the request 'id'). They are also broadcasted, which
may not make sense to other clients for various commands (all the dump
& query for example) and this prevent from adding meaningful client
'id' field (since it may conflict with other clients 'id').

I propose to re-introduce a better 'async' support, where command may
return with the request 'id' only to the caller, when they are
finished, and not block the monitor. This will be opt-out, so existing
code do not have to change.  Furthermore, since reply order is not
guaranteed, clients will have to have the new 'async' capability to
use those async commands. This scheme is not incompatible with having
additional commands to query the command status, or broadcast related
events to all clients.

Marc-André Lureau (36):
  qapi: add comment block before ChardevDummy
  qapi: add missing @
  qapi: add some line spacing to help doc parser
  monitor: use qapi for qmp_capabilities command
  qapi: move examples to json schema
  qapi: move documentation bits in schema files
  qapi: add some headings in docs
  qapi: add qapi2texi script
  qapi: remove qmp-events.txt
  texi2pod: learn quotation, deftp and deftypefn
  build-sys: generate QAPI doc based on json
  build-sys: generate qmp-commands.txt
  build-sys: do not generate qmp-commands-old.h
  monitor: remove usage of generated marshal functions
  monitor: register gen:false commands manually
  qmp: register qapi commands (no middle mode)
  qmp: use qmp_dispatch()
  qapi: remove "middle" mode
  qmp: implement qmp_query_commands without qmp_cmds
  qmp: remove old qmp-commands table
  misc: spelling
  qmp: teach qmp_dispatch() to take a pre-filled QDict
  qmp: use a return callback for the command reply
  qmp: add QmpClient
  qmp: introduce async command type
  qmp: check that async command have an 'id'
  scripts: learn 'async' qapi commands
  scripts: ensure -async commands are declared async
  qapi: take 'id' from request
  tests: change /0.15/* tests to /qmp/*
  tests: add /qmp/dispatch_cmd_async test
  qmp: update qmp-spec about async capability
  monitor: add 'async' capability
  console: graphic_hw_update return true if async
  console: add graphic_hw_update_done()
  console: add screendump-async

 Makefile                                |   43 +-
 Makefile.target                         |    7 +-
 docs/qmp/qmp-events.txt                 |  664 -----
 docs/qmp/qmp-spec.txt                   |   17 +-
 hw/display/qxl-render.c                 |   14 +-
 hw/display/qxl.c                        |    8 +-
 hw/display/qxl.h                        |    2 +-
 include/qapi/qmp/dispatch.h             |   29 +-
 include/ui/console.h                    |    4 +-
 monitor.c                               |  409 +--
 qapi-schema.json                        | 1258 ++++++++-
 qapi/block-core.json                    |  456 +++-
 qapi/block.json                         |   48 +-
 qapi/common.json                        |   37 +-
 qapi/event.json                         |  211 ++
 qapi/introspect.json                    |    2 +-
 qapi/qmp-dispatch.c                     |   92 +-
 qapi/qmp-registry.c                     |   26 +-
 qapi/rocker.json                        |   46 +
 qapi/trace.json                         |   14 +
 qga/main.c                              |   26 +-
 qga/qapi-schema.json                    |    2 +
 qmp-commands.hx                         | 4324 -------------------------------
 scripts/qapi-commands.py                |  156 +-
 scripts/qapi-introspect.py              |    7 +-
 scripts/qapi.py                         |  103 +-
 scripts/qapi2texi.py                    |  293 +++
 scripts/texi2pod.pl                     |   44 +-
 tests/Makefile                          |    1 +
 tests/qapi-schema/async.err             |    0
 tests/qapi-schema/async.exit            |    1 +
 tests/qapi-schema/async.json            |    1 +
 tests/qapi-schema/async.out             |    5 +
 tests/qapi-schema/qapi-schema-test.json |    2 +
 tests/qapi-schema/qapi-schema-test.out  |    5 +
 tests/qapi-schema/test-qapi.py          |    6 +-
 tests/test-qmp-commands.c               |  132 +-
 ui/console.c                            |   77 +-
 vl.c                                    |    1 +
 39 files changed, 3108 insertions(+), 5465 deletions(-)
 delete mode 100644 docs/qmp/qmp-events.txt
 delete mode 100644 qmp-commands.hx
 create mode 100755 scripts/qapi2texi.py
 create mode 100644 tests/qapi-schema/async.err
 create mode 100644 tests/qapi-schema/async.exit
 create mode 100644 tests/qapi-schema/async.json
 create mode 100644 tests/qapi-schema/async.out

-- 
2.4.3