Re: [Qemu-devel] [PATCH v9 07/17] qapi: Start converting to new qapi union layout

[Markus Armbruster]

Eric Blake <address@hidden> writes:

> We have two issues with our qapi union layout:
> 1) Even though the QMP wire format spells the tag 'type', the
> C code spells it 'kind', requiring some hacks in the generator.
> 2) The C struct uses an anonymous union, which places all tag
> values in the same namespace as all non-variant members. This
> leads to spurious collisions if a tag value matches a QMP name.
>
> This patch is the front end for a series that converts to a
> saner qapi union layout.  By the end of the series, we will no
> longer have the type/kind mismatch, and all tag values will be
> under a named union, which requires clients to access
> 'obj->u.value' instead of 'obj->value'.  But since the
> conversion touches a number of files, it is easiest if we
> temporarily support BOTH layouts simultaneously.
>
> Given a simple union qapi type:
>
> { 'union':'Foo', 'data': { 'a':'int', 'b':'bool' } }
>
> we make the following changes in generated qapi-types.h:
>
> | struct Foo {
> |-    FooKind kind;
> |-    union { /* union tag is @kind */
> |+    union {
> |+        FooKind kind;
> |+        FooKind type;
> |+    };
> |+    union { /* union tag is @type */
> |         void *data;
> |         int64_t a;
> |         bool b;
> |+        union { /* union tag is @type */
> |+            void *data;
> |+            int64_t a;
> |+            bool b;
> |+        } u;
> |     };
> | };

This is clever and ugly in equal measure.  I respect that.  Fortunately,
it's also temporary.

> Flat unions do not need the anonymous union for the tag member,
> as we already fixed that to use the member name instead of 'kind'
> back in commit 0f61af3e.

Unlike then, we need multiple commits for simple unions, because they're
more widely used?

>                           On the other hand, the duplication
> means that we temporarily cannot support 'u' as a branch name.

Separate paragraph, because now you're talking about the *other*
anonymous union.

> Later, when the conversions are complete, we will remove the
> duplication hacks and restore support for 'u' as a branch name.
>
> Note, however, that we do not rename the generated enum, which
> is still 'FooKind'.  A further patch could generate implicit
> enums as 'FooType', but that causes more churn to C code, and
> gets harder since the generator already reserved the '*Kind'
> namespace, but there are already QMP constructs with '*Type'
> naming which means we cannot easily reserve it for qapi.

Oh, we can reserve whatever we want in QAPI, it's just a lot of churn to
adapt the QAPI-using code.

I'd simply say "but that would cause substantial churn to C code, as
there are already QAPI definitions with '*Type' naming".

> Signed-off-by: Eric Blake <address@hidden>
>
> ---
> v9: new patch, but incorporates parts of v5 31/46 and Markus' RFC:
> http://lists.gnu.org/archive/html/qemu-devel/2015-10/msg02236.html
> ---
>  scripts/qapi-types.py                   | 26 +++++++++++++++++++-------
>  scripts/qapi-visit.py                   | 24 +++++++++---------------
>  tests/qapi-schema/qapi-schema-test.json |  4 +++-
>  tests/qapi-schema/qapi-schema-test.out  |  4 ++--
>  4 files changed, 33 insertions(+), 25 deletions(-)
>

First part: generate C structs as described in the commit message.

> diff --git a/scripts/qapi-types.py b/scripts/qapi-types.py
> index bcef39d..0a14451 100644
> --- a/scripts/qapi-types.py
> +++ b/scripts/qapi-types.py
> @@ -136,11 +136,23 @@ struct %(c_name)s {
>      if base:
>          ret += gen_struct_fields([], base)
>      else:
> +        # TODO As a hack, we emit both 'kind' and 'type'. Ultimately, we
> +        # want to use only 'type', but the conversion is large enough to
> +        # require staging over several commits.
>          ret += mcgen('''
> -    %(c_type)s kind;
> +    union {
> +        %(c_type)s kind;
> +        %(c_type)s type;
> +    };
>  ''',
>                       c_type=c_name(variants.tag_member.type.name))
>
> +    # TODO As a hack, we emit the union twice, once as an anonymous union
> +    # and once as a named union.  Ultimately, we want to use only the
> +    # named union version (as it avoids conflicts between tag values as
> +    # branch names competing with non-variant QMP names), but the conversion
> +    # is large enough to require staging over several commits.
> +    tmp = ''
>      # FIXME: What purpose does data serve, besides preventing a union that
>      # has a branch named 'data'? We use it in qapi-visit.py to decide
>      # whether to bypass the switch statement if visiting the discriminator
> @@ -149,25 +161,25 @@ struct %(c_name)s {
>      # should not be any data leaks even without a data pointer.  Or, if
>      # 'data' is merely added to guarantee we don't have an empty union,
>      # shouldn't we enforce that at .json parse time?
> -    ret += mcgen('''
> +    tmp += mcgen('''
>      union { /* union tag is @%(c_name)s */
>          void *data;
>  ''',
> -                 # TODO ugly special case for simple union
> -                 # Use same tag name in C as on the wire to get rid of
> -                 # it, then: c_name=c_name(variants.tag_member.name)
> -                 c_name=c_name(variants.tag_name or 'kind'))
> +                 c_name=c_name(variants.tag_member.name))
>
>      for var in variants.variants:
>          # Ugly special case for simple union TODO get rid of it
>          typ = var.simple_union_type() or var.type
> -        ret += mcgen('''
> +        tmp += mcgen('''
>          %(c_type)s %(c_name)s;
>  ''',
>                       c_type=typ.c_type(),
>                       c_name=c_name(var.name))
>
> +    ret += tmp
> +    ret += '    ' + '\n    '.join(tmp.split('\n'))
>      ret += mcgen('''
> +    } u;
>      };
>  };
>  ''')

It took me some head-scratching to understand why this generates
correctly indented output.  If it wasn't temporary code, I'd ask for
cleanup.

Second part: convert qapi-visit.py.  Not mentioned in commit message.
Separate patch, perhaps?

> diff --git a/scripts/qapi-visit.py b/scripts/qapi-visit.py
> index 91bf350..2afe811 100644
> --- a/scripts/qapi-visit.py
> +++ b/scripts/qapi-visit.py
> @@ -182,18 +182,18 @@ void visit_type_%(c_name)s(Visitor *v, %(c_name)s 
> **obj, const char *name, Error
>      if (err) {
>          goto out;
>      }
> -    visit_get_next_type(v, (int*) &(*obj)->kind, %(c_name)s_qtypes, name, 
> &err);
> +    visit_get_next_type(v, (int*) &(*obj)->type, %(c_name)s_qtypes, name, 
> &err);
>      if (err) {
>          goto out_obj;
>      }
> -    switch ((*obj)->kind) {
> +    switch ((*obj)->type) {
>  ''',
>                  c_name=c_name(name))
>
>      for var in variants.variants:
>          ret += mcgen('''
>      case %(case)s:
> -        visit_type_%(c_type)s(v, &(*obj)->%(c_name)s, name, &err);
> +        visit_type_%(c_type)s(v, &(*obj)->u.%(c_name)s, name, &err);
>          break;
>  ''',
>                       case=c_enum_const(variants.tag_member.type.name,
> @@ -255,22 +255,16 @@ void visit_type_%(c_name)s(Visitor *v, %(c_name)s 
> **obj, const char *name, Error
>      visit_type_%(c_type)s(v, &(*obj)->%(c_name)s, "%(name)s", &err);
>  ''',
>                       c_type=variants.tag_member.type.c_name(),
> -                     # TODO ugly special case for simple union
> -                     # Use same tag name in C as on the wire to get rid of
> -                     # it, then: c_name=c_name(variants.tag_member.name)
> -                     c_name='kind',
> +                     c_name=c_name(variants.tag_member.name),
>                       name=variants.tag_member.name)
>      ret += gen_err_check(label='out_obj')
>      ret += mcgen('''
> -    if (!visit_start_union(v, !!(*obj)->data, &err) || err) {
> +    if (!visit_start_union(v, !!(*obj)->u.data, &err) || err) {
>          goto out_obj;
>      }
>      switch ((*obj)->%(c_name)s) {
>  ''',
> -                 # TODO ugly special case for simple union
> -                 # Use same tag name in C as on the wire to get rid of
> -                 # it, then: c_name=c_name(variants.tag_member.name)
> -                 c_name=c_name(variants.tag_name or 'kind'))
> +                 c_name=c_name(variants.tag_member.name))
>
>      for var in variants.variants:
>          # TODO ugly special case for simple union
> @@ -282,13 +276,13 @@ void visit_type_%(c_name)s(Visitor *v, %(c_name)s 
> **obj, const char *name, Error
>                                         var.name))
>          if simple_union_type:
>              ret += mcgen('''
> -        visit_type_%(c_type)s(v, &(*obj)->%(c_name)s, "data", &err);
> +        visit_type_%(c_type)s(v, &(*obj)->u.%(c_name)s, "data", &err);
>  ''',
>                           c_type=simple_union_type.c_name(),
>                           c_name=c_name(var.name))
>          else:
>              ret += mcgen('''
> -        visit_type_implicit_%(c_type)s(v, &(*obj)->%(c_name)s, &err);
> +        visit_type_implicit_%(c_type)s(v, &(*obj)->u.%(c_name)s, &err);
>  ''',
>                           c_type=var.type.c_name(),
>                           c_name=c_name(var.name))
> @@ -304,7 +298,7 @@ out_obj:
>      error_propagate(errp, err);
>      err = NULL;
>      if (*obj) {
> -        visit_end_union(v, !!(*obj)->data, &err);
> +        visit_end_union(v, !!(*obj)->u.data, &err);
>      }
>      error_propagate(errp, err);
>      err = NULL;

Third part: work around temporary clash with 'u'.  Needs to remain in
this patch, obviously.  Suggest to amend the commit message to say

    On the other hand, the duplication means that we temporarily cannot
    support 'u' as a branch name.  Adapt a few tests that do.

> diff --git a/tests/qapi-schema/qapi-schema-test.json 
> b/tests/qapi-schema/qapi-schema-test.json
> index 22e15eb..876ce18 100644
> --- a/tests/qapi-schema/qapi-schema-test.json
> +++ b/tests/qapi-schema/qapi-schema-test.json
> @@ -113,8 +113,10 @@
>  # should still be valid as a type or union branch name. And although
>  # '*Kind' and '*List' are forbidden as type names, they should not be
>  # forbidden as a member or branch name.
> +# TODO - we temporarily do not support 'u' as branch name, while converting
> +# code to use the new union layout
>  { 'struct': 'has_a', 'data': { 'MyKind': 'int', 'MyList': ['int'] } }
> -{ 'union': 'u', 'data': { 'u': 'uint8', 'myKind': 'has_a',
> +{ 'union': 'u', 'data': { 'u8': 'uint8', 'myKind': 'has_a',
>                            'myList': 'has_a' } }
>
>  # testing commands
> diff --git a/tests/qapi-schema/qapi-schema-test.out 
> b/tests/qapi-schema/qapi-schema-test.out
> index feaf20d..cb12435 100644
> --- a/tests/qapi-schema/qapi-schema-test.out
> +++ b/tests/qapi-schema/qapi-schema-test.out
> @@ -202,10 +202,10 @@ object has_a
>      member MyKind: int optional=False
>      member MyList: intList optional=False
>  object u
> -    case u: :obj-uint8-wrapper
> +    case u8: :obj-uint8-wrapper
>      case myKind: :obj-has_a-wrapper
>      case myList: :obj-has_a-wrapper
> -enum uKind ['u', 'myKind', 'myList']
> +enum uKind ['u8', 'myKind', 'myList']
>  command user_def_cmd None -> None
>     gen=True success_response=True
>  command user_def_cmd1 :obj-user_def_cmd1-arg -> None