[PATCH 0/8] Inter-plugin interactions with QPP

[Andrew Fasano]

Hello,

This is a series of patches expanding the TCG plugin system to add what we're
calling the "QEMU Plugin-to-Plugin (QPP)" interface that allows for
interactions between TCG plugins. This is a follow up to the Sept 1 2022 RFC
"Support interactions between TCG plugins." The goal of this interface is to
enable plugins to expand on other plugins and reduce code duplication. This
patch series includes documentation, tests and commented code, but a
high-level summary is below along with a discussion of the current
implementation, how it would affect plugin developers, and a summary of the
changes since the original RFC.

**Summary**
The QPP interface allows two types of interactions between plugins:
1) Exported functions: A plugin may wish to allow other plugins to call
one of the functions it has defined. To do this, the plugin must mark
the function definition as publicly visible with the QEMU_PLUGIN_EXPORT
macro and place a definition in an included header file using the
QPP_FUN_PROTOTYPE macro. Other plugins can then include this header and
call the exported function by combining the name of the target plugin
with the name of the exported function and "_qpp" at the end.
For example, consider a hypothetical plugin that collects a list of
cache misses. This plugin could export two functions using the QPP
interface: one to allow another plugin to query this list and another
to empty the list. This would enable the development of another plugin
that examines guest CPU state to identify process changes and reports
the cache misses per process. With the QPP interface, this second plugin
would not need to duplicate any logic from the first.

2) Callbacks: Multiple plugins may wish to take some action when some
event of interest occurs inside a running guest. To support modularity
and reduce code duplication, the QPP callback system allows this logic
to be contained in single plugin that detects whenever a given event
occurs and exposes a callback with a given name. Another plugin can then
request to have one of its own functions run whenever this event occurs.
Additional plugins could also use this same callback to run additional
logic whenever this event occurs.

For example, consider (again) a hypothetical plugin that detects when
the current guest process changes by analyzing the guest CPU state. This
plugin could define a callback named "on_process_change" and trigger
this callback event whenever it detects a process change. Other plugins
could then be developed that take various actions on process changes by
registering internal functions to run on this event.
These patches and examples are inspired by the PANDA project
(https://panda.re and https://github.com/panda-re/panda), a fork of QEMU
modified to support dynamic program analysis and reverse engineering.
PANDA also includes a large plugin system with a similar interface for
interactions between plugins. We are a part of the PANDA team and have
seen how the ability for plugins to interact with other plugins reduces
code duplication and enables the creation of many useful plugins.

**Implementation Overview**
These patches increment the TCG plugin version from 1 to 2 and require
plugins of v2 or greater to specify a plugin name. During plugin load, an
error is raised if two plugins are loaded that use the same name.
Using these unique names, plugins can import functions from other
plugins and register local functions to be run on callback events
managed by other plugins. To accomplish this, these patches add
the following functions to the core plugin code:
 - qemu_plugin_import_function: this is used by the QPP_FUN_PROTOTYPE
   macro to find and resolve the handle of the function to be
   imported given the target plugin name and function name.
 - qemu_plugin_create_callback: this can be called by plugins to
   create a new callback using the unique plugin id and the callback
   name.
 - qemu_plugin_run_callback: this is called by plugins to define when
   registered callback functions get run for the named callback and
   specify the args that are passed to those functions.
 - qemu_plugin_reg_callback: this can be used by plugins to register
   functions on another plugin's defined callback.
 - qemu_plugin_unreg_callback: this removes a previously registered
   function from the callback's list of registered functions if it
   exists there.

The QPP implementation is mostly contained within the api, but we
did add a helpful macro for qemu_plugin_import_function inside a
header file plugin-qpp.h called QPP_FUN_PROTOTYPE. This macro
must be used after defining the qemu_plugin_name variable (would
love to change this part of the design if anyone has better ideas).

The QPP_FUN_PROTOYPE macro enables a plugin to expose a function
it defines to other plugins. This macro enables a plugin to provide a single
header file that both defines the exported functions for the plugin
that provides them as well as providing a definition and auto-resolving the
function pointers for other plugins that wish to call them. This macro is used
in a header file that is included by both the plugin that defines a function as
well as plugins that wish to use the function. This prototype creates a
constructor function that runs on plugin load. If the target plugin
name differs from the value of the current plugin name, this function
will use qemu_plugin_import_function to resolve the target function in
that plugin. If this fails, it will print a warning and abort.

Finally, these patches include a pair of test plugins, qpp_srv and
qpp_client. The former provides a trivial QPP callback which simply runs
when the plugin is unloaded as well as two functions to add and subtract
integers. The latter of these plugins registers a function to run
on this callback and uses the two exported functions. These plugins are
found in tests/plugin_qpp and have been integrated into the make check-tcg
tests.

**Changes Since the 9/1/22 RFC**
Based on the feedback we received from Alex, we changed our design to use
the QEMU plugin APIs instead of allowing plugins to directly dlopen and
interact with one another.
We bumped the QPP version number and mandated that developers specify a unique
name for each plugin instead of using the source filenames as the plugin names.
We moved our pair of example plugins to the tests directory and configured
them to be run and tested as part of the check-tcg tests.

**Next steps**
We are working on building some more useful example plugins that builds on this
API. Pending feedback on this patch series, we would be happy to either merge
these API changes on their own or hold off until such these example plugins
are ready.

We have developed some plugins already, but our initial examples depend on
some poorly-written APIs we've added to access guest registers and memory.
Those APIs aren't ready for upstreaming yet, which is why we aren't including
those patches and example plugins in this patch series. However the code is
available in our tree if anyone is interested:
  - Register accessing: 
https://github.com/panda-re/qemu/commit/b97c5a56edd0ba3b5f6ab16bf531ac1f7abaac04
  - Memory accessing: 
https://github.com/panda-re/qemu/commit/dba9d1ceffeeeddf2b5232d64ca97147a4f5a3a5

Using the QPP system from this patch series and the ability to access guest
registers and memory, we developed have a set of three of plugins that provide
OS introspection, syscall detection, and logging such that on every system
call in a Linux guest we can record the system call number and
information on the current running process. These generate output like:
  Syscall at pc 7f0b815478cd: number 3: Process 'init', pid 211, ppid 1
  Syscall at pc 7fff775feb58: number 228: Process 'systemd-udevd', pid 137, 
ppid 1

As our register and memory access APIs aren't yet ready to merge, we
haven't included these plugins as a part of this patch series, but their
code is also available in our tree:
  - Linux OS Introspection: 
https://github.com/panda-re/qemu/blob/ppp_beta_with_plugins/contrib/plugins/osi_linux/
  - Syscall detection: 
https://github.com/panda-re/qemu/blob/ppp_beta_with_plugins/contrib/plugins/syscalls.c
  - Syscall logging with process details: 
https://github.com/panda-re/qemu/blob/ppp_beta_with_plugins/contrib/plugins/syscalls_logger.c

We plan to explore integrating this with qemu's existing user-mode syscall
tracing system and would love to eventually merge these plugins if there's
interest and the necessary APIs (QPP + register/memory access) become
available.

We welcome any feedback for this patch series and the next steps we're
interested in pursuing.
Thank you,
Andrew Fasano & Elysia Witham

Elysia Witham (8):
  docs/devel: describe QPP API
  plugins: version 2, require unique plugin names
  plugins: add id_to_plugin_name
  plugins: add core API functions for QPP callbacks
  plugins: implement QPP callbacks
  plugins: implement QPP import function
  include/qemu: added macro for QPP import function
  tests: build and run QPP tests

 docs/devel/tcg-plugins.rst                |  91 ++++++++++++++-
 include/qemu/plugin-qpp.h                 |  54 +++++++++
 include/qemu/qemu-plugin.h                |  68 ++++++++++-
 plugins/api.c                             | 132 ++++++++++++++++++++++
 plugins/core.c                            |  47 ++++++++
 plugins/loader.c                          |  61 +++++++++-
 plugins/plugin.h                          |  24 ++++
 plugins/qemu-plugins.symbols              |   5 +
 tests/Makefile.include                    |   2 +-
 tests/meson.build                         |   1 +
 tests/plugin/bb.c                         |   1 +
 tests/plugin/empty.c                      |   1 +
 tests/plugin/insn.c                       |   1 +
 tests/plugin/mem.c                        |   1 +
 tests/plugin/syscall.c                    |   1 +
 tests/plugin_qpp/meson.build              |   7 ++
 tests/plugin_qpp/qpp_client.c             |  32 ++++++
 tests/plugin_qpp/qpp_srv.c                |  37 ++++++
 tests/plugin_qpp/qpp_srv.h                |  12 ++
 tests/tcg/Makefile.target                 |  29 +++++
 tests/tcg/aarch64/Makefile.softmmu-target |   2 +
 tests/tcg/arm/Makefile.softmmu-target     |   1 +
 tests/tcg/arm/Makefile.target             |   2 +
 tests/tcg/multiarch/Makefile.target       |   2 +
 24 files changed, 605 insertions(+), 9 deletions(-)
 create mode 100644 include/qemu/plugin-qpp.h
 create mode 100644 tests/plugin_qpp/meson.build
 create mode 100644 tests/plugin_qpp/qpp_client.c
 create mode 100644 tests/plugin_qpp/qpp_srv.c
 create mode 100644 tests/plugin_qpp/qpp_srv.h

-- 
2.34.1