[Qemu-devel] [PATCH RFC v3 00/32] qapi: QMP introspection

[Markus Armbruster]

Still RFC, because it has a couple of FIXMEs and TODOs.

* PATCH 01-24 basically replace the interemediate representation.  The
  replacement isn't complete, but fully functional.  See PATCH 02 for
  rationale and future work.

  PATCH 06-09 and 16-17 are temporary scaffolding to ease review.  I
  don't intend to include them in future non-RFC revisions.

* PATCH 25-29 replace the '**' type bypass by a proper top type called
  'any'.

* PATCH 30 is the introspection RFC.

* PATCH 31-32 are minor variations to hide a bit more detail from
  clients.

I'm afraid the size of my series will strain reviewers.  I did my
level best to split things up into reviewable pieces.

Series is based on "[PATCH 00/26] qapi: Another round of fixes and
cleanups".  It also needs my "[PATCH for-2.4 0/2] qom: Fix misuse of
Error API" to pass tests.

You can fetch it from my git://repo.or.cz/qemu/armbru.git branch
qapi-introspect.

v3:
* Straightforward rebase, R-bys retained unless noted otherwise.  Due
  to my widespread changes, you may prefer to review from scratch
  anyway.
* Old PATCH 01-19 split off into separate non-RFC series "[PATCH
  00/26] qapi: Another round of fixes and cleanups"
* PATCH 02:
  - Commit message spelling fix [Eric] TODO
  - Clarify patch's non-effect in commit message.
  - Assert enumeration constants are unique.
  - Assert object's base is an object without variants [Eric].
  - Drop a few superfluous check() recursions.
  - Use arg_type, ret_type instead of args, ret for commands [Eric],
    and arg_type instead of data for events.
  - Lower simple variants to flat ones as described on
    qapi-code-gen.txt.  Replace QAPISchemaObjectType's .flat by
    .simple_union_type(), for use by pre-existing code-generation
    warts.
  - _make_implicit_object_type() idempotent (previous item needs
    that).
  - Clean up how empty command argument dictionaries work [Eric].
  - Same for event.
  - No R-by to drop.
* PATCH 03:
  - Fix QAPISchemaEnumType.c_null() for empty enums [Eric].
  - R-by dropped.
* PATCH 04:
  - Use arg_type, ret_type instead of args, ret for commands [Eric],
    and arg_type instead of data for events.
* PATCH 05:
  - Use arg_type, ret_type instead of args, ret for commands [Eric],
    and arg_type instead of data for events.
  - .flat is gone, drop code to print it.
  - Expected test output changed due to revision of PATCH 02.
* PATCH 06-09,16-17: New, replacing old PATCH 25 "qapi: Make
  generators work on sorted schema expressions".
* PATCH 10:
  - Update qapi-code-gen.txt with the patch's changes.
  - Earlier patches ripple into gen_union().
  - Nicer comment on -b [Eric].
  - Objects with both members and unions aren't implemented.  Add an
    assertion [Eric].
  - No R-by to drop.
* PATCH 11:
  - Earlier patches ripple into gen_visit_union().
  - Rename guard QAPI_VISIT_BUILTIN_VISITOR_DECL to
    QAPI_VISIT_BUILTIN.
  - Nicer comment on -b [Eric].
  - Objects with both members and unions aren't implemented.  Add an
    assertion [Eric].
  - No R-by to drop.
* PATCH 12:
  - Use arg_type, ret_type instead of args, ret [Eric].
  - Zap .regy unconditionally in visit_end (squashed from PATCH 30).
* PATCH 13:
  - Commit message fixed and improved [Eric].
  - Update qapi-code-gen.txt with the patch's changes.
* PATCH 15
  - Copyright notice pasto fixed [Eric].
  - Minimize patch by delaying more renames to PATCH 19.
  - Use arg_type instead of data.
  - Free .event_names in visit_end(), moved here from PATCH 30.
* PATCH 18:
  - Moved; was right after PATCH 12 before.
  - One hunk dropped due to "[PATCH 25/26] qapi-commands: Drop useless
    initialization".
* PATCH 19:
  - Use arg_type, ret_type instead of args, ret for commands [Eric],
    and arg_type instead of data or params for events.
  - Consistently use single quotes for strings holding generated code
    [Eric].
  - Generate just 'FOO' instead of 'struct FOO' when possible [Eric].
  - Earlier patches ripple into gen_union() and gen_visit_union().
  - R-by dropped (death by a thousand cuts).
* PATCH 20:
  - Straightforward conflicts due to new "[PATCH 14/26] qapi: Document
    that input visitor semantics are prone to leaks" and "[PATCH
    15/26] qapi: Document shortcoming with union 'data' branch".
* PATCH 21:
  - Use arg_type, ret_type instead of args, ret [Eric].
  - Straightforward conflict due to "[PATCH 25/26] qapi-commands: Drop
    useless initialization".
* PATCH 22:
  - Rename in qmp.c's comment, too [Eric].
* PATCH 23:
  - Use arg_type, ret_type instead of args, ret for commands [Eric],
    and arg_type instead of data for events.
  - Consistently use single quotes for strings holding generated code
    [Eric].
* PATCH 24:
  - Update qapi-code-gen.txt with the patch's changes.
  - Use arg_type, ret_type instead of args, ret [Eric].
  - Free .visited_ret_types in visit_end(), moved here from PATCH 30.
* PATCH 26:
  - Indentation fix [Eric].
  - Some test case updates moved here from PATCH 29: move positive
    test case from args-returns-any.json to qapi-schema-test.json
    (with new qmp_guest_sync() dummy in test-qmp-commands.c), move
    negative test case to args-any.json, drop bogus test case.
  - Rename flat-union-base-star.* to flat-union-base-any.*.
  - Test QMP input visitor's visit_type_any() doesn't add extra
    members [Eric].
* PATCH 27:
  - Drop a TODO obsoleted by the patch [Eric].
* PATCH 28:
  - Old PATCH 42+43 squashed together [Eric].
* PATCH 29:
  - Documentation update on 'gen' was botched, rephrase [Eric].
  - Some test case updates move to PATCH 26.
* PATCH 30:
  - success-response isn't implementation detail, add TODO and fix
    commit message [Eric].
  - Cover new qapi-introspect.py in qapi-code-gen.txt [Eric].
  - Clarify qmp_query_schema()'s comment.
  - Fix introspect.json's comment on default values [Eric].
  - Replace SchemaInfoObjectVariant member 'members' by 'type' [Eric].
  - New SchemaInfoAlternateMember [Eric].
  - Use arg_type, ret_type instead of args, ret for commands [Eric],
    and arg_type instead of data for events.
  - Rewrite actual code generation: instead of generating a list of
    JSON strings directly, first generate a list of dicts, then format
    them as JSON.
  - Fix to use " instead of ' for JSON strings [Eric].
  - Make the generated array const [Eric].
  - Fix accidental drop of a line of code [Eric].
  - Fix file name in tests/.gitignore [Eric].
  - Revision of PATCH 02 ripples into test output diffs.
  - Move new test from test-qmp-input-visitor.c to
    test-qmp-input-strict.c to make it catch extra members.
  - R-by dropped.
* PATCH 32:
  - Update qapi-code-gen.txt with the patch's changes.
  - Rewrite actual code generation: instead of generating a list of
    JSON strings directly, first generate a list of dicts, then format
    them as JSON.
  - Use plain integers as type names [Eric].
  - Don't hide names of builtin types [Eric].
  - Put a TODO on type name comments in commit message.

Markus Armbruster (32):
  qapi: Rename class QAPISchema to QAPISchemaParser
  qapi: New QAPISchema intermediate reperesentation
  qapi: QAPISchema code generation helper methods
  qapi: New QAPISchemaVisitor
  tests/qapi-schema: Convert test harness to QAPISchemaVisitor
  qapi: Split up some typedefs to ease review
  qapi: Generate comments to simplify splitting for review
  Revert "qapi: Generate comments to simplify splitting for review"
  Revert "qapi: Split up some typedefs to ease review"
  qapi-types: Convert to QAPISchemaVisitor, fixing flat unions
  qapi-visit: Convert to QAPISchemaVisitor, fixing bugs
  qapi-commands: Convert to QAPISchemaVisitor
  qapi: De-duplicate enum code generation
  qapi-event: Eliminate global variable event_enum_value
  qapi-event: Convert to QAPISchemaVisitor, fixing data with base
  qapi: Generate comments to simplify splitting for review
  Revert "qapi: Generate comments to simplify splitting for review"
  qapi: Replace dirty is_c_ptr() by method c_null()
  qapi: Clean up after recent conversions to QAPISchemaVisitor
  qapi-visit: Rearrange code a bit
  qapi-commands: Rearrange code
  qapi: Rename qmp_marshal_input_FOO() to qmp_marshal_FOO()
  qapi: De-duplicate parameter list generation
  qapi-commands: De-duplicate output marshaling functions
  qapi: Improve built-in type documentation
  qapi: Introduce a first class 'any' type
  qom: Don't use 'gen': false for qom-get, qom-set, object-add
  qapi-schema: Fix up misleading specification of netdev_add
  qapi: Pseudo-type '**' is now unused, drop it
  qapi: New QMP command query-schema for QMP schema introspection
  qapi-introspect: Map all integer types to 'int'
  qapi-introspect: Hide type names

 .gitignore                                         |   1 +
 Makefile                                           |   9 +-
 Makefile.objs                                      |   4 +-
 docs/qapi-code-gen.txt                             | 154 +++--
 docs/writing-qmp-commands.txt                      |   8 +-
 include/monitor/monitor.h                          |   3 -
 include/qapi/visitor-impl.h                        |   2 +
 include/qapi/visitor.h                             |   1 +
 monitor.c                                          |  17 +-
 qapi-schema.json                                   |  25 +-
 qapi/introspect.json                               |  71 +++
 qapi/qapi-dealloc-visitor.c                        |   9 +
 qapi/qapi-visit-core.c                             |   6 +
 qapi/qmp-input-visitor.c                           |  11 +
 qapi/qmp-output-visitor.c                          |   9 +
 qmp-commands.hx                                    | 266 +++++----
 qmp.c                                              |  27 +-
 scripts/qapi-commands.py                           | 272 ++++-----
 scripts/qapi-event.py                              | 239 +++-----
 scripts/qapi-introspect.py                         | 197 +++++++
 scripts/qapi-types.py                              | 355 +++++-------
 scripts/qapi-visit.py                              | 350 +++++-------
 scripts/qapi.py                                    | 634 +++++++++++++++++----
 tests/.gitignore                                   |   1 +
 tests/Makefile                                     |  15 +-
 tests/qapi-schema/alternate-good.out               |  16 +-
 tests/qapi-schema/args-any.err                     |   1 +
 .../{type-bypass-no-gen.exit => args-any.exit}     |   0
 tests/qapi-schema/args-any.json                    |   2 +
 .../qapi-schema/{type-bypass.err => args-any.out}  |   0
 tests/qapi-schema/args-member-array.out            |  14 +-
 tests/qapi-schema/comments.out                     |   5 +-
 tests/qapi-schema/empty.out                        |   4 +-
 tests/qapi-schema/enum-empty.out                   |   5 +-
 tests/qapi-schema/event-case.out                   |   5 +-
 tests/qapi-schema/flat-union-base-any.err          |   1 +
 ...ion-base-star.exit => flat-union-base-any.exit} |   0
 ...ion-base-star.json => flat-union-base-any.json} |   2 +-
 ...e-bypass-no-gen.out => flat-union-base-any.out} |   0
 tests/qapi-schema/flat-union-base-star.err         |   1 -
 tests/qapi-schema/flat-union-base-star.out         |   0
 tests/qapi-schema/flat-union-reverse-define.out    |  22 +-
 tests/qapi-schema/ident-with-escape.out            |   8 +-
 tests/qapi-schema/include-relpath.out              |   5 +-
 tests/qapi-schema/include-repetition.out           |   5 +-
 tests/qapi-schema/include-simple.out               |   5 +-
 tests/qapi-schema/indented-expr.out                |   8 +-
 tests/qapi-schema/qapi-schema-test.json            |   8 +-
 tests/qapi-schema/qapi-schema-test.out             | 219 +++++--
 tests/qapi-schema/returns-int.out                  |   6 +-
 tests/qapi-schema/test-qapi.py                     |  38 +-
 tests/qapi-schema/type-bypass-no-gen.err           |   1 -
 tests/qapi-schema/type-bypass-no-gen.json          |   2 -
 tests/qapi-schema/type-bypass.exit                 |   1 -
 tests/qapi-schema/type-bypass.json                 |   2 -
 tests/qapi-schema/type-bypass.out                  |   3 -
 tests/test-qmp-commands.c                          |   5 +
 tests/test-qmp-input-strict.c                      |  57 +-
 tests/test-qmp-input-visitor.c                     |  49 +-
 tests/test-qmp-output-visitor.c                    |  53 ++
 60 files changed, 2054 insertions(+), 1185 deletions(-)
 create mode 100644 qapi/introspect.json
 create mode 100644 scripts/qapi-introspect.py
 create mode 100644 tests/qapi-schema/args-any.err
 rename tests/qapi-schema/{type-bypass-no-gen.exit => args-any.exit} (100%)
 create mode 100644 tests/qapi-schema/args-any.json
 rename tests/qapi-schema/{type-bypass.err => args-any.out} (100%)
 create mode 100644 tests/qapi-schema/flat-union-base-any.err
 rename tests/qapi-schema/{flat-union-base-star.exit => 
flat-union-base-any.exit} (100%)
 rename tests/qapi-schema/{flat-union-base-star.json => 
flat-union-base-any.json} (95%)
 rename tests/qapi-schema/{type-bypass-no-gen.out => flat-union-base-any.out} 
(100%)
 delete mode 100644 tests/qapi-schema/flat-union-base-star.err
 delete mode 100644 tests/qapi-schema/flat-union-base-star.out
 delete mode 100644 tests/qapi-schema/type-bypass-no-gen.err
 delete mode 100644 tests/qapi-schema/type-bypass-no-gen.json
 delete mode 100644 tests/qapi-schema/type-bypass.exit
 delete mode 100644 tests/qapi-schema/type-bypass.json
 delete mode 100644 tests/qapi-schema/type-bypass.out

-- 
2.4.3