Re: [Qemu-devel] RFC: libyajl for JSON

[Markus Armbruster]

Eric Blake <address@hidden> writes:

> Loaded question in response to
> https://lists.gnu.org/archive/html/qemu-devel/2015-10/msg06988.html, but

Discussion of our parser's enormous appetite for wasting RAM.  Fixable,
but it's work, and it's not its only defect.

> posting as a new thread to call attention to it:
>
> Libvirt uses libyajl to parse and format JSON. Would it be worth
> dragging in yet another prerequisite library into qemu and reuse
> libyajl's parser instead of our own hand-rolled one?
>
> I know that a while ago, the answer was "as long as we support
> out-of-the-box builds on RHEL 5, that platform lacks yajl therefore we
> can't depend on it" (and libvirt's solution on RHEL 5 is "qemu doesn't
> support QMP and thus doesn't use JSON and thus libvirt doesn't need yajl
> there").
>
> But now that we have just recently bumped the minimum glib and python
> versions to something not available on RHEL 5, it may also be time to
> start thinking about outsourcing to libyajl, because as far as I can
> tell, every platform that currently supports qemu out of the box has a
> version of libyajl. And since libvirt has already figured out the grunt
> work of how to simultaneously code to both the 1.x and 2.x APIs, it's
> not that much of a stretch to reuse that work in qemu.

I'm not particularly attached to having our very own JSON parser.
However:

> On the other hand, one of the "features" of qemu's hand-rolled json
> parser is the ability to do qobject_from_jsonf("{'foo':%s}", "bar")
> (that is, we extended JSON with our notion of single-quoted strings, and
> with our notion of %s and similar escape sequences for piecing together
> multiple inputs into a single input stream without having to first
> g_strdup_printf our pieces into a single string).  I don't know if
> libyajl lets us add extensions to the language it parses.

Actually two separate extensions:

* Single-quoted strings

  The extension's purpose is avoiding quotes in C.  Example:

      "{ 'execute': 'migrate_set_speed', 'arguments': { 'value': 10 } }"

  is easier to read than

      "{ \"execute\": \"migrate_set_speed\", \"arguments\": { \"value\": 10 } }"

  Most actual uses are in tests.

  JSON5, a proposed extension to JSON also supports single-quoted
  strings.  So does Python.

* Optional interpolation

  If you pass a va_list to the parser, it replaces certain escape
  sequences by values from that va_list.  The escape sequences are a
  subset of printf conversion specifiers, to enable compile-time
  checking with __attribute__((format...)).

  We used to rely on this for creating "rich" error objects, but those
  are long gone (commit df1e608).  Perhaps two dozen uses are left.

  We could convert them to use "normal" interpolation, i.e. first format
  the arguments as JSON, then interpolate with g_strdup_printf() or
  similar, then parse.  Formatting only to parse it right back is
  inelegant.  Also inefficient, but that probably doesn't matter here.

  The current interface could be retained as convenience function to
  interpolate and feed the result to the JSOn parser.

  Alternatively, we could build the QObject manually instead.  More
  efficient than either kind of interpolation, but a good deal less
  readable.

Got one more, actually a pitfall, not an extension:

* Representation of JSON numbers

  RFC 7159 doesn't specify how numbers are to be represented.

  Many JSON implementations represent any JSON number as double.  This
  can represent signed integers up to 53 bits exactly.

  Our parser represents numbers as int64_t when possible, else as
  double.

  Unlike the extensions above, this one is essential: any parser that
  can't do it is useless for us.  Can yajl do it?

More extensions or pitfalls might be lurking in our parser.  Therefore,
replacing our parser by a suitable library is not without risk.  I guess
we could do it over a full development cycle.  No way for 2.5.

If we decide to attempt replacing it in the next development cycle, we
might want to refrain from fixing its bugs except for true show
stoppers.