[Qemu-devel] [PATCH RFC 00/21] Modularize generated QAPI code

[Markus Armbruster]

Our qapi-schema.json is composed of modules connected by include
directives, but the generated code is monolithic all the same: one
qapi-types.h with all the types, one qapi-visit.h with all the
visitors, and so forth.  These monolithic headers get included all
over the place.  In my "build everyhing" tree, adding a QAPI type
recompiles about 4500 out of 4800 objects.

Nobody would write such monolithic headers by hand.  It stands to
reason that one shouldn't generate them, either.

This series' basic idea is to split up generated headers to mirror the
schema's modular structure: one header per module.  That way, you can
include just what you need.

The series is RFC for a number of reasons:

* The split is implemented only for qapi-types.h.  That one should
  provide the biggest benefits, though.

* There's a bit of code duplication.

* I haven't re-read my patches, yet.

Even in this incomplete state, the compile-time improvements can be
massive.  Before this series, any QAPI schema change recompiles some
4500 out of 4800 objects in my "build everything" tree.  Afterwards,
adding a type to qapi/migration.json recompiles less than 400, adding
a QMP event recompiles less than 200, and a documentation change no
longer recompiles anything.

Related: Marc-André's 'unit' pragma proposal.  That's a different way
to split off parts of the generated code, motivated by the desire to
use poisoned identifiers such as TARGET_I386.  I noted in my review of
v3 that I "can either accept it, or come up with a better solution."
This is my attempt at a better solution.  It's a bit more ambitious,
and thus more useful (I hope).  The pragma has one theoretical
advantage, though: you can modularize the generated output in
different ways than the input.  The patches using don't do that,
however.

Based-on: <address@hidden>
[PATCH v3 00/19] Clean up includes to reduce compile time

Markus Armbruster (21):
  qapi: Streamline boilerplate comment generation
  qapi: Generate up-to-date copyright notice
  qapi: New classes QAPIGenC, QAPIGenH, QAPIGenDoc
  qapi: Reduce use of global variables in generators some
  qapi: Turn generators into modules
  qapi-gen: New common driver for code and doc generators
  qapi: Move parse_command_line() next to its only use
  qapi: Touch generated files only when they change
  qapi: Don't absolutize include file name in error messages
  qapi/common: Eliminate QAPISchema.exprs
  qapi: Lift error reporting from QAPISchema.__init__() to callers
  qapi: Concentrate QAPISchemaParser.exprs updates in .__init__()
  qapi: Record 'include' directives in parse tree
  qapi: Generate in source order
  qapi: Record 'include' directives in intermediate representation
  qapi/types qapi/visit: Make visitors use QAPIGen more
  qapi/types qapi/visit: Generate built-in stuff into separate files
  qapi/common: Fix guardname() for funny filenames
  qapi/types: Generate separate .h, .c for each module
  Include less of qapi-types.h
  qapi: Empty out qapi-schema.json

 Makefile                                           |  131 +-
 Makefile.objs                                      |   20 +-
 crypto/cipherpriv.h                                |    2 +-
 hw/block/block.c                                   |    1 +
 hw/block/hd-geometry.c                             |    1 +
 hw/net/rocker/rocker_fp.c                          |    2 +-
 include/block/block.h                              |    2 +-
 include/block/dirty-bitmap.h                       |    2 +-
 include/chardev/char.h                             |    1 +
 include/crypto/cipher.h                            |    2 +-
 include/crypto/hash.h                              |    2 +-
 include/crypto/hmac.h                              |    2 +-
 include/crypto/secret.h                            |    1 +
 include/crypto/tlscreds.h                          |    1 +
 include/hw/block/block.h                           |    2 +-
 include/hw/block/fdc.h                             |    2 +-
 include/hw/ppc/spapr_drc.h                         |    1 +
 include/hw/qdev-properties.h                       |    2 +
 include/io/dns-resolver.h                          |    1 +
 include/migration/colo.h                           |    2 +-
 include/migration/failover.h                       |    2 +-
 include/migration/global_state.h                   |    1 +
 include/monitor/monitor.h                          |    1 +
 include/net/filter.h                               |    1 +
 include/net/net.h                                  |    2 +-
 include/qapi/error.h                               |    2 +-
 include/qapi/qmp/qobject.h                         |    2 +-
 include/qapi/visitor.h                             |    2 +-
 include/qemu/sockets.h                             |    2 +-
 include/qemu/throttle.h                            |    2 +-
 include/qom/cpu.h                                  |    1 +
 include/qom/object.h                               |    2 +-
 include/sysemu/arch_init.h                         |    2 +-
 include/sysemu/balloon.h                           |    2 +-
 include/sysemu/dump.h                              |    2 +
 include/sysemu/hostmem.h                           |    1 +
 include/sysemu/replay.h                            |    2 +
 include/sysemu/sysemu.h                            |    1 +
 include/sysemu/tpm.h                               |    1 +
 include/sysemu/watchdog.h                          |    2 +-
 include/ui/input.h                                 |    2 +-
 migration/migration.h                              |    1 +
 migration/ram.h                                    |    2 +-
 net/tap_int.h                                      |    2 +-
 qapi-schema.json                                   | 3098 +-------------------
 qapi/misc.json                                     | 3090 +++++++++++++++++++
 qapi/run-state.json                                |   10 +
 replication.h                                      |    1 +
 scripts/qapi-gen.py                                |   95 +
 scripts/qapi/__init__.py                           |    0
 scripts/{qapi-commands.py => qapi/commands.py}     |   98 +-
 scripts/{qapi.py => qapi/common.py}                |  290 +-
 scripts/{qapi2texi.py => qapi/doc.py}              |   28 +-
 scripts/{qapi-event.py => qapi/events.py}          |  101 +-
 scripts/{qapi-introspect.py => qapi/introspect.py} |   87 +-
 scripts/{qapi-types.py => qapi/types.py}           |  218 +-
 scripts/{qapi-visit.py => qapi/visit.py}           |  208 +-
 tests/Makefile.include                             |   56 +-
 tests/qapi-schema/comments.out                     |    3 +-
 tests/qapi-schema/doc-bad-section.out              |    5 +-
 tests/qapi-schema/doc-good.out                     |   33 +-
 tests/qapi-schema/doc-good.texi                    |    3 +-
 tests/qapi-schema/empty.out                        |    2 +-
 tests/qapi-schema/event-case.out                   |    3 +-
 tests/qapi-schema/ident-with-escape.out            |    7 +-
 tests/qapi-schema/include-no-file.err              |    2 +-
 tests/qapi-schema/include-relpath.out              |    7 +-
 tests/qapi-schema/include-repetition.out           |   12 +-
 tests/qapi-schema/include-simple.out               |    5 +-
 tests/qapi-schema/indented-expr.out                |    3 +-
 tests/qapi-schema/qapi-schema-test.out             |  321 +-
 tests/qapi-schema/test-qapi.py                     |   17 +-
 ui/vnc.h                                           |    1 +
 73 files changed, 4040 insertions(+), 3984 deletions(-)
 create mode 100644 qapi/misc.json
 create mode 100755 scripts/qapi-gen.py
 create mode 100644 scripts/qapi/__init__.py
 rename scripts/{qapi-commands.py => qapi/commands.py} (78%)
 rename scripts/{qapi.py => qapi/common.py} (93%)
 rename scripts/{qapi2texi.py => qapi/doc.py} (93%)
 mode change 100755 => 100644
 rename scripts/{qapi-event.py => qapi/events.py} (69%)
 rename scripts/{qapi-introspect.py => qapi/introspect.py} (78%)
 rename scripts/{qapi-types.py => qapi/types.py} (54%)
 rename scripts/{qapi-visit.py => qapi/visit.py} (66%)

-- 
2.13.6