[Qemu-devel] [RFC][PATCH v7 00/16] virtagent: host/guest communication agent

[Michael Roth]

These patches apply to master (3-07-2011), and can also be obtained from:
git://repo.or.cz/qemu/mdroth.git virtagent_v7

CHANGES IN V7:

 - Removed dependency on xmlrpc-c for data transport. Now using JSON via QEMU's 
qjson qobject<->json conversion routines. Binary encoding mechanisms such as 
Protocol Buffers and ASN.1/BER were considered, but due to limited library 
support, and limitations of isa/virtio serial transport that would have 
required an additional layer of encoding to reliably determine RPC boundaries 
during transport (more here: 
http://www.mail-archive.com/address@hidden/msg56237.html), qobject<->json 
seemed to be the most prudent route.
 - Logic to handle management/scheduling of bi-directional RPCs is now 
decoupled from transport layer for readability and better support for future 
additions such as session-level agents and threaded execution of RPCs.
 - Added thorough documentation of virtagent protocol to virtagent-manager.h
 - Improved documentation for RPCs
 - Fixes for guest agent lockfile handling
 - Removed viewdmesg/viewfile, will be replacing these with a more robust 
getfile RPC/command shortly
 - Workaround for guests that, in certain situations, fail to retrieve a pushed 
buffer from virtqueue in timely fashion (Rusty submitted patch several months 
ago, don't know the status)
 - Added guest agent support for switching isa-serial ports to RAW mode 
(previously reliant on socat intermediary)
 - Added tagging of RPCs to allow for multiple in-flight requests
 - qemu-va guest agent no longer builds as part of host tools target
 - Various other bug fixes and cleanups

CHANGES IN V6:

 - Added a sentinel value to reliably detect the start of an "http" hdr. Used 
to skip past partially sent http content from previous "sessions"
 - Added http hdr tag (currently hardcoded for testing, will switch to uuid) to 
filter out valid-but-unexpected content in channel from previous "sessions"
 - Added timeout mechanism to avoid hanging monitor when agent isn't running
 - Added timed back-off on read's from a virtio-serial that result in ret=0 to 
avoid spinning if host isn't connected. 
 - Added daemonize flags to qemu-va
 - Added sane defaults for channel type and virtio-serial port path
 - Various bug fixes for state machine/job handling logic

CHANGES IN V5:

 - Dependency on virtproxy dropped, virtagent now handles transport and 
multiplexing of bi-directional RPCs internally
 - Removed duplification of qemu_set_fd_handler()-centered i/o code. Support 
for interacting with objects that use qemu_set_fd_handler() now available to 
tools via qemu-tools.c and a set of generalized utility functions
 - Fixed memory leaks in client/monitor functions
 - Various cleanups

CHANGES IN V4:

 - Added guest agent capabilities negotiation
 - Added RPC/monitor command to invoke guest shutdown/reboot/powerdown
 - Added RPC/monitor command to the guest agent
 - Added guest startup notification ("hello")
 - Added syslog()'ing of guest agent RPCs
 - Various cleanups

CHANGES IN V3:

 - Integrated virtagent server into virtproxy chardev. Usage examples below.
 - Consolidated RPC server/client setup into a pair of init routines
 - Fixed buffer overflow in agent_viewfile() and various memory leaks

CHANGES IN V2:

 - All RPC communication is now done using asynchronous/non-blocking read/write 
handlers
 - Previously fork()'d RPC server loop is now integrated into qemu-vp/virtproxy 
i/o loop
 - Cleanups/suggestions from previous RFC

OVERVIEW:

There are a wide range of use cases motivating the need for a guest agent of 
some sort to extend the functionality/usability/control offered by QEMU. Some 
examples include graceful guest shutdown/reboot and notifications thereof, 
copy/paste syncing between host/guest, guest statistics gathering, file access, 
etc.

Ideally these would all be served by a single, easilly extensible agent that 
can be deployed in a wide range of guests. Virtagent is an JSON RPC server, 
integrated into QEMU and a simple guest daemon, aimed at providing this type of 
functionality.

DESIGN:

There are actually 2 RPC servers:

1) a server in the guest agent which handles RPC requests from QEMU
2) a server in the host, integrated into the virtagent chardev, to handle RPC 
requests sent by the guest agent (mainly for handling asynchronous events 
reported by the agent).

Communication is done via RPCs (JSON/HTTP between host and guest), albeit with 
a non-standard implementation that allows for multiplexing server/client RPC 
over a single virtio/isa serial channel.

EXAMPLE USAGE:

 - Build into host:
    ./configure --target-list=x86_64-softmmu --enable-io-thread
    make

 - Build guest agent (in guest):
    ./configure --target-list=x86_64-softmmu --enabled-io-thread
    make qemu-va

 - Configure guest agent to talk to host via virtio-serial
    # start guest with virtio-serial/virtagent. for example (RHEL6):
    qemu \
    -chardev virtagent,id=test0 \
    -device virtio-serial \
    -device virtserialport,chardev=test0,name=virtagent0 \
    -monitor stdio
    ...
    # in the guest:
    sudo ./qemu-va -c virtio-serial -p /dev/virtio-ports/virtagent0
    ...
    # monitor commands
    (qemu) va_ping
    status: success
    (qemu) va_viewfile /proc/meminfo
    MemTotal:        3985488 kB
    MemFree:          400524 kB
    Buffers:          220556 kB
    Cached:          2073160 kB
    SwapCached:            0 kB
    ...
    Hugepagesize:       2048 kB
    DirectMap4k:        8896 kB
    DirectMap2M:     4110336 kB
    (qemu) va_shutdown powerdown
    (qemu)

KNOWN ISSUES/PLANS:
 - Implement RPCs for stateful open/read/close of guest files.
 - Implement RPC for guest script/command execution
 - Scan for sentinel value while reading http content as well to immediately 
detect truncated requests/responses and avoid accidentally consuming new ones
 - switch to standard logging/trace mechanisms
 - the client socket that qemu connects to send RPCs is a hardcoded filepath. 
This is unacceptable as the socket is channel/process specific and things will 
break when multiple guests are started.


 Makefile              |    4 +-
 Makefile.objs         |    2 +-
 Makefile.target       |    2 +-
 cpus.c                |   83 +-------
 hmp-commands.hx       |   48 +++++
 monitor.c             |    1 +
 qemu-char.c           |   44 ++++
 qemu-char.h           |    4 +
 qemu-ioh.c            |  210 ++++++++++++++++++++
 qemu-ioh.h            |   43 ++++
 qemu-tool.c           |  115 +++++++++++-
 qemu-tool.h           |   26 +++
 qemu-va.c             |  247 +++++++++++++++++++++++
 qerror.c              |    8 +
 qerror.h              |    6 +
 qmp-commands.hx       |   97 +++++++++
 roms/seabios          |    2 +-
 virtagent-common.c    |  206 +++++++++++++++++++
 virtagent-common.h    |   95 +++++++++
 virtagent-manager.c   |  326 ++++++++++++++++++++++++++++++
 virtagent-manager.h   |  130 ++++++++++++
 virtagent-server.c    |  387 ++++++++++++++++++++++++++++++++++++
 virtagent-server.h    |   40 ++++
 virtagent-transport.c |  432 ++++++++++++++++++++++++++++++++++++++++
 virtagent.c           |  524 +++++++++++++++++++++++++++++++++++++++++++++++++
 virtagent.h           |   51 +++++
 vl.c                  |   86 ++-------
 27 files changed, 3071 insertions(+), 148 deletions(-)