[Qemu-devel] [PATCH v2 2/2] full introspection support for QMP

[Amos Kong]

Introduces new monitor command to query QMP schema information,
the return data is a dynamical and nested dict/list, it contains
the useful metadata to help management to check feature support,
QMP commands detail, etc.

I added a document for QMP introspection support.
(docs/qmp-full-introspection.txt)

We need to parse all commands json definition, and generated a
dynamical tree, QMP infrastructure will convert the tree to
json string and return to QMP client.

So here I defined a 'DataObject' type in qapi-schema.json,
it's used to describe the dynamical dictionary/list/string.

{ 'type': 'DataObject',
  'data': { '*key': 'str', '*type': 'str', '*data': ['DataObject'] } }

Not all the keys in data will be used.
 # List: type
 # Dict: key, type
 # nested List: type, data
 # nested Dict: key, type, data

The DataObject is described in docs/qmp-full-introspection.txt in
detail.

The following content gives an example of query-tpm-types:

 ## Define example in qapi-schema.json:

 { 'enum': 'TpmType', 'data': [ 'passthrough' ] }
 { 'command': 'query-tpm-types', 'returns': ['TpmType'] }

 ## Returned description:

 {
     "name": "query-tpm-types",
     "type": "Command",
     "returns": [
         {
             "type": "TpmType",
             "data": [
                 {
                     "type": "passthrough"
                 }
             ]
         }
     ]
 },

'TpmType' is a defined type, it will be extended in returned
description. [ 'passthrough' ] is a list, so 'type' and 'data'
will be used.

TODO:
We can add events definations to qapi-schema.json by another
patch, then event can also be queried.

Introduce another command 'query-qga-schema' to query QGA schema
information, it's easy to add this support with current current
patch.

Signed-off-by: Amos Kong <address@hidden>
---
 docs/qmp-full-introspection.txt | 143 +++++++++++++++++++
 qapi-schema.json                |  69 +++++++++
 qmp-commands.hx                 |  39 +++++
 qmp.c                           | 307 ++++++++++++++++++++++++++++++++++++++++
 4 files changed, 558 insertions(+)
 create mode 100644 docs/qmp-full-introspection.txt

diff --git a/docs/qmp-full-introspection.txt b/docs/qmp-full-introspection.txt
new file mode 100644
index 0000000..cc0fb80
--- /dev/null
+++ b/docs/qmp-full-introspection.txt
@@ -0,0 +1,143 @@
+= full introspection support for QMP =
+
+== Purpose ==
+
+Add a new interface to provide QMP schema information to management,
+the return data is a dynamical and nested dict/list, it contains
+the useful metadata to help management to check feature support,
+QMP commands detail, etc.
+
+== Usage ==
+
+Execute QMP command:
+
+  { "execute": "query-qmp-schema" }
+
+Returns:
+
+  { "return": [
+      {
+          "name": "query-name",
+          "type": "Command",
+          "returns": [
+              {
+                  "key": "*name",
+                  "type": "str"
+              }
+          ]
+      },
+      ...
+   }
+
+The whole schema information will be returned in one go, it contains
+commands and event. It doesn't support to be filtered by type or name.
+
+We have four types (ImageInfo, BlockStats, PciDeviceInfo, SchemaData)
+that uses themself in their own define data directly or indirectly,
+we will not repeatedly extend them to avoid dead loop.
+
+== more description of 'DataObject' type ==
+
+We use 'DataObject' to describe dynamical data struct, it might be
+nested dictionary, list or string.
+
+'DataObject' itself is a arbitrary and nested dictionary, the
+dictionary has three keys ('key', 'type', 'data'), 'key' and
+'data' are optional.
+
+* For describing Dictionary, we set the key to 'key', and set the
+  value to 'type'
+* For describing List, we don't set 'key', just set the value to
+  'type'
+* If the value of dictionary or list is non-native type, we extend
+  the non-native type to dictionary, set it to 'data',  and set the
+  non-native type's name to 'type'.
+* If the value of dictionary or list is dictionary or list, 'type'
+  won't be set.
+
+== examples ==
+
+1) Dict, value is native type
+{ 'id': 'str', ... }
+--------------------
+[
+    {
+        "key": "id",
+        "type": "str"
+    },
+    .....
+]
+
+2) Dict, value is defined types
+{ 'options': 'TpmTypeOptions' }
+-------------------------------
+[
+    {
+        "key": "options",
+        "type": "TpmTypeOptions",
+        "data": [
+            {
+                "key": "passthrough",
+                "type": "str",
+            }
+        ]
+    },
+    .....
+]
+
+3) List, value is native type
+['str', ... ]
+-------------
+[
+    {
+        "type": "str"
+    },
+    ....
+]
+
+4) List, value is defined types
+['TpmTypeOptions', ... ]
+------------------------
+[
+    {
+        "type": "TpmTypeOptions",
+        "data": [
+            {
+                "key": "passthrough",
+                "type": "str",
+            }
+        ]
+    },
+    .....
+]
+
+5) Dict, value is dictionary
+{ 'info': { 'age': 'init', ... } }
+-----------------------------
+[
+    {
+        "key": "info",
+        "data": [
+            {
+                "key": "age",
+                "type": "init",
+            },
+            ...
+        ]
+    },
+]
+
+6) Dict, value is list
+{ 'info': [ 'str', ... } }
+-----------------------------
+[
+    {
+        "key": "info",
+        "data": [
+            {
+                "type": "str",
+            },
+            ...
+        ]
+    },
+]
diff --git a/qapi-schema.json b/qapi-schema.json
index 7b9fef1..cf03391 100644
--- a/qapi-schema.json
+++ b/qapi-schema.json
@@ -3679,3 +3679,72 @@
             '*cpuid-input-ecx': 'int',
             'cpuid-register': 'X86CPURegister32',
             'features': 'int' } }
+
+##
+# @DataObject
+#
+# Details of a data object, it can be nested dictionary/list
+#
+# @key: #optional Data object key
+#
+# @type: Data object type name
+#
+# @optional: #optional whether the data object is optional
+#
+# @data: #optional DataObject list, can be a dictionary or list type
+#
+# Since: 1.6
+##
+{ 'type': 'DataObject',
+  'data': { '*key': 'str', '*type': 'str', '*optional': 'bool', '*data': 
['DataObject'] } }
+
+##
+# @SchemaMetaType
+#
+# Possible meta types of a schema entry
+#
+# @Command: QMP command
+#
+# @Type: defined new data type
+#
+# @Enumeration: enumeration data type
+#
+# @Union: union data type
+#
+# Since: 1.6
+##
+{ 'enum': 'SchemaMetaType',
+  'data': ['Command', 'Type', 'Enumeration', 'Union'] }
+
+##
+# @SchemaEntry
+#
+# Details of schema items
+#
+# @type: Entry's type in string format
+#
+# @name: Entry name
+#
+# @data: #optional list of DataObject. This can have different meaning
+#        depending on the 'type' value. For example, for a QMP command,
+#        this member contains an argument listing. For an enumeration,
+#        it contains the enum's values and so on
+#
+# @returns: #optional list of DataObject, return data after executing
+#           QMP command
+#
+# Since: 1.6
+##
+{ 'type': 'SchemaEntry', 'data': { 'type': 'SchemaMetaType',
+  'name': 'str', '*data': ['DataObject'], '*returns': ['DataObject'] } }
+
+##
+# @query-qmp-schema
+#
+# Query QMP schema information
+#
+# Returns: list of @SchemaEntry. Returns an error if json string is invalid.
+#
+# Since: 1.6
+##
+{ 'command': 'query-qmp-schema', 'returns': ['SchemaEntry'] }
diff --git a/qmp-commands.hx b/qmp-commands.hx
index e075df4..e3cbe93 100644
--- a/qmp-commands.hx
+++ b/qmp-commands.hx
@@ -3047,3 +3047,42 @@ Example:
 <- { "return": {} }
 
 EQMP
+
+    {
+        .name       = "query-qmp-schema",
+        .args_type  = "",
+        .mhandler.cmd_new = qmp_marshal_input_query_qmp_schema,
+    },
+
+
+SQMP
+query-qmp-schema
+----------------
+
+query qmp schema information
+
+Return a json-object with the following information:
+
+- "name": qmp schema name (json-string)
+- "type": qmp schema type, it can be 'comand', 'type', 'enum', 'union', 'event'
+- "data": schema data (json-object, optional)
+- "returns": return data of qmp command (json-object, optional)
+
+Example:
+
+-> { "execute": "query-qmp-schema" }
+<- { "return": [
+       {
+           "name": "query-name",
+           "type": "Command",
+           "returns": [
+               {
+                    "key": "*name",
+                   "type": "str"
+               }
+           ]
+       }
+     ]
+   }
+
+EQMP
diff --git a/qmp.c b/qmp.c
index 4c149b3..3ace3a6 100644
--- a/qmp.c
+++ b/qmp.c
@@ -25,6 +25,8 @@
 #include "sysemu/blockdev.h"
 #include "qom/qom-qobject.h"
 #include "hw/boards.h"
+#include "qmp-schema.h"
+#include "qapi/qmp/qjson.h"
 
 NameInfo *qmp_query_name(Error **errp)
 {
@@ -486,6 +488,311 @@ CpuDefinitionInfoList *qmp_query_cpu_definitions(Error 
**errp)
     return arch_query_cpu_definitions(errp);
 }
 
+/*
+ * Use a string to record the visit path, type index of each node
+ * will be saved to the string, indexes are split by ':'.
+ */
+static char visit_path_str[1024];
+
+/* push the type index to visit_path_str */
+static void push_id(int id)
+{
+    char *end = strrchr(visit_path_str, ':');
+    char type_idx[256];
+    int num;
+
+    num = sprintf(type_idx, "%d:", id);
+
+    if (end) {
+        /* avoid overflow */
+        assert(end - visit_path_str + 1 + num < sizeof(visit_path_str));
+        sprintf(end + 1, "%d:", id);
+    } else {
+        sprintf(visit_path_str, "%d:", id);
+    }
+}
+
+/* pop the type index from visit_path_str */
+static void pop_id(void)
+{
+    char *p = strrchr(visit_path_str, ':');
+
+    assert(p != NULL);
+    *p = '\0';
+    p = strrchr(visit_path_str, ':');
+    if (p) {
+        *(p + 1) = '\0';
+    } else {
+        visit_path_str[0] = '\0';
+    }
+}
+
+static const char *qstring_copy_str(QObject *data)
+{
+    QString *qstr;
+
+    if (!data) {
+        return NULL;
+    }
+    qstr = qobject_to_qstring(data);
+    if (qstr) {
+        return qstring_get_str(qstr);
+    } else {
+        return NULL;
+    }
+}
+
+static DataObjectList *visit_qobj_dict(QObject *data);
+static DataObjectList *visit_qobj_list(QObject *data);
+
+/* extend defined type to json object */
+static DataObjectList *extend_type(const char* str)
+{
+    DataObjectList *data_list;
+    QObject *data;
+    QDict *qdict;
+    const QDictEntry *ent;
+    int i;
+
+    /* don't extend builtin types */
+    if (!strcmp(str, "str") || !strcmp(str, "int") ||
+        !strcmp(str, "number") || !strcmp(str, "bool") ||
+        !strcmp(str, "int8") || !strcmp(str, "int16") ||
+        !strcmp(str, "int32") || !strcmp(str, "int64") ||
+        !strcmp(str, "uint8") || !strcmp(str, "uint16") ||
+        !strcmp(str, "uint32") || !strcmp(str, "uint64")) {
+        return NULL;
+    }
+
+    for (i = 0; qmp_schema_table[i]; i++) {
+        data = qobject_from_json(qmp_schema_table[i]);
+        assert(data != NULL);
+
+        qdict = qobject_to_qdict(data);
+        assert(qdict != NULL);
+
+        ent = qdict_first(qdict);
+        if (!qdict_get(qdict, "enum") && !qdict_get(qdict, "type")
+            && !qdict_get(qdict, "union")) {
+            continue;
+        }
+
+        if (!strcmp(str, qstring_copy_str(ent->value))) {
+            char *start, *end;
+            char cur_idx[256];
+            char type_idx[256];
+
+            start = visit_path_str;
+            sprintf(type_idx, "%d", i);
+            while(start) {
+                end = strchr(start, ':');
+                if (!end) {
+                    break;
+                }
+                snprintf(cur_idx, end - start + 1, "%s", start);
+                start = end + 1;
+                /* if the type was already extended in parent node,
+                 * we don't extend it again to avoid dead loop. */
+                if (!strcmp(cur_idx, type_idx)) {
+                    return NULL;
+                }
+            }
+            /* push index to visit_path_str before extending */
+            push_id(i);
+
+            data = qdict_get(qdict, "data");
+            if(data) {
+                if (data->type->code == QTYPE_QDICT) {
+                    data_list = visit_qobj_dict(data);
+                } else if (data->type->code == QTYPE_QLIST) {
+                    data_list = visit_qobj_list(data);
+                }
+                /* pop index from visit_path_str after extending */
+                pop_id();
+
+                return data_list;
+            }
+        }
+    }
+
+    return NULL;
+}
+
+static DataObjectList *visit_qobj_list(QObject *data)
+{
+    DataObjectList *obj_list, *obj_last_entry, *obj_entry;
+    DataObject *obj_info;
+    const QListEntry *ent;
+    QList *qlist;
+
+    qlist = qobject_to_qlist(data);
+    assert(qlist != NULL);
+
+    obj_list = NULL;
+    for (ent = qlist_first(qlist); ent; ent = qlist_next(ent)) {
+        obj_info = g_malloc0(sizeof(*obj_info));
+        obj_entry = g_malloc0(sizeof(*obj_entry));
+        obj_entry->value = obj_info;
+        obj_entry->next = NULL;
+
+        if (!obj_list) {
+            obj_list = obj_entry;
+        } else {
+            obj_last_entry->next = obj_entry;
+        }
+        obj_last_entry = obj_entry;
+
+        if (ent->value->type->code == QTYPE_QDICT) {
+            obj_info->data = visit_qobj_dict(ent->value);
+        } else if (ent->value->type->code == QTYPE_QLIST) {
+            obj_info->data = visit_qobj_list(ent->value);
+        } else if (ent->value->type->code == QTYPE_QSTRING) {
+            obj_info->has_type = true;
+            obj_info->type = g_strdup(qstring_copy_str(ent->value));
+            obj_info->data = extend_type(qstring_copy_str(ent->value));
+        }
+        if (obj_info->data) {
+            obj_info->has_data = true;
+        }
+    }
+
+    return obj_list;
+}
+
+static DataObjectList *visit_qobj_dict(QObject *data)
+{
+    DataObjectList *obj_list, *obj_last_entry, *obj_entry;
+    DataObject *obj_info;
+    const QDictEntry *ent;
+    QDict *qdict;
+
+    qdict = qobject_to_qdict(data);
+    assert(qdict != NULL);
+
+    obj_list = NULL;
+    for (ent = qdict_first(qdict); ent; ent = qdict_next(qdict, ent)) {
+        obj_info = g_malloc0(sizeof(*obj_info));
+        obj_entry = g_malloc0(sizeof(*obj_entry));
+        obj_entry->value = obj_info;
+        obj_entry->next = NULL;
+
+        if (!obj_list) {
+            obj_list = obj_entry;
+        } else {
+            obj_last_entry->next = obj_entry;
+        }
+        obj_last_entry = obj_entry;
+
+        if (ent->key[0] == '*') {
+            obj_info->key = g_strdup(ent->key + 1);
+            obj_info->has_optional = true;
+            obj_info->optional = true;
+        } else {
+            obj_info->key = g_strdup(ent->key);
+        }
+        obj_info->has_key = true;
+
+        if (ent->value->type->code == QTYPE_QDICT) {
+            obj_info->data = visit_qobj_dict(ent->value);
+        } else if (ent->value->type->code == QTYPE_QLIST) {
+           obj_info->data = visit_qobj_list(ent->value);
+        } else if (ent->value->type->code == QTYPE_QSTRING) {
+            obj_info->has_type = true;
+            obj_info->type = g_strdup(qstring_copy_str(ent->value));
+            obj_info->data = extend_type(qstring_copy_str(ent->value));
+        }
+        if (obj_info->data) {
+            obj_info->has_data = true;
+        }
+    }
+
+    return obj_list;
+}
+
+SchemaEntryList *qmp_query_qmp_schema(Error **errp)
+{
+    SchemaEntryList *list, *last_entry, *entry;
+    SchemaEntry *info;
+    DataObjectList *obj_entry;
+    DataObject *obj_info;
+    QObject *data;
+    QDict *qdict;
+    int i;
+
+    list = NULL;
+    for (i = 0; qmp_schema_table[i]; i++) {
+        data = qobject_from_json(qmp_schema_table[i]);
+        assert(data != NULL);
+
+        qdict = qobject_to_qdict(data);
+        assert(qdict != NULL);
+
+        if (qdict_get(qdict, "command")) {
+            info = g_malloc0(sizeof(*info));
+            info->type = SCHEMA_META_TYPE_COMMAND;
+            info->name = strdup(qdict_get_str(qdict, "command"));
+        } else {
+            continue;
+        }
+
+        memset(visit_path_str, 0, sizeof(visit_path_str));
+        data = qdict_get(qdict, "data");
+        if (data) {
+            info->has_data = true;
+            if (data->type->code == QTYPE_QLIST) {
+                info->data = visit_qobj_list(data);
+            } else if (data->type->code == QTYPE_QDICT) {
+                info->data = visit_qobj_dict(data);
+            } else if (data->type->code == QTYPE_QSTRING) {
+                info->data = 
extend_type(qstring_get_str(qobject_to_qstring(data)));
+                if (!info->data) {
+                    obj_info = g_malloc0(sizeof(*obj_info));
+                    obj_entry = g_malloc0(sizeof(*obj_entry));
+                    obj_entry->value = obj_info;
+                    obj_info->has_type = true;
+                    obj_info->type = g_strdup(qdict_get_str(qdict, "data"));
+                    info->data = obj_entry;
+                }
+            } else {
+                abort();
+            }
+        }
+
+        memset(visit_path_str, 0, sizeof(visit_path_str));
+        data = qdict_get(qdict, "returns");
+        if (data) {
+            info->has_returns = true;
+            if (data->type->code == QTYPE_QLIST) {
+                info->returns = visit_qobj_list(data);
+            } else if (data->type->code == QTYPE_QDICT) {
+                info->returns = visit_qobj_dict(data);
+            } else if (data->type->code == QTYPE_QSTRING) {
+                info->returns = extend_type(qstring_copy_str(data));
+                if (!info->returns) {
+                    obj_info = g_malloc0(sizeof(*obj_info));
+                    obj_entry = g_malloc0(sizeof(*obj_entry));
+                    obj_entry->value = obj_info;
+                    obj_info->has_type = true;
+                    obj_info->type = g_strdup(qdict_get_str(qdict, "returns"));
+                    info->returns = obj_entry;
+                }
+            }
+        }
+
+        entry = g_malloc0(sizeof(DataObjectList *));
+        entry->value = info;
+        entry->next = NULL;
+        if (!list) {
+            list = entry;
+        } else {
+            last_entry->next = entry;
+        }
+        last_entry = entry;
+    }
+
+    return list;
+}
+
 void qmp_add_client(const char *protocol, const char *fdname,
                     bool has_skipauth, bool skipauth, bool has_tls, bool tls,
                     Error **errp)
-- 
1.8.3.1