[Qemu-devel] [RFC][PATCH v6 00/23] virtagent: host/guest RPC communication agent

[Michael Roth]

These patches apply to master (1-14-2011), and can also be obtained from:
git://repo.or.cz/qemu/mdroth.git virtagent_v6

CHANGES IN V6:

 - Added a sentinel value to reliably detect the start of an "http" hdr. Used 
to skip past partially sent http content from previous "sessions"
 - Added http hdr tag (currently hardcoded for testing, will switch to uuid) to 
filter out valid-but-unexpected content in channel from previous "sessions"
 - Added timeout mechanism to avoid hanging monitor when agent isn't running
 - Added timed back-off on read's from a virtio-serial that result in ret=0 to 
avoid spinning if host isn't connected. 
 - Added daemonize flags to qemu-va
 - Added sane defaults for channel type and virtio-serial port path
 - Various bug fixes for state machine/job handling logic

CHANGES IN V5:

 - Dependency on virtproxy dropped, virtagent now handles transport and 
multiplexing of bi-directional RPCs internally
 - Removed duplification of qemu_set_fd_handler()-centered i/o code. Support 
for interacting with objects that use qemu_set_fd_handler() now available to 
tools via qemu-tools.c and a set of generalized utility functions
 - Fixed memory leaks in client/monitor functions
 - Various cleanups

CHANGES IN V4:

 - Added guest agent capabilities negotiation
 - Added RPC/monitor command to invoke guest shutdown/reboot/powerdown
 - Added RPC/monitor command to the guest agent
 - Added guest startup notification ("hello")
 - Added syslog()'ing of guest agent RPCs
 - Various cleanups

CHANGES IN V3:

 - Integrated virtagent server into virtproxy chardev. Usage examples below.
 - Consolidated RPC server/client setup into a pair of init routines
 - Fixed buffer overflow in agent_viewfile() and various memory leaks

CHANGES IN V2:

 - All RPC communication is now done using asynchronous/non-blocking read/write 
handlers
 - Previously fork()'d RPC server loop is now integrated into qemu-vp/virtproxy 
i/o loop
 - Cleanups/suggestions from previous RFC

OVERVIEW:

There are a wide range of use cases motivating the need for a guest agent of 
some sort to extend the functionality/usability/control offered by QEMU. Some 
examples include graceful guest shutdown/reboot and notifications thereof, 
copy/paste syncing between host/guest, guest statistics gathering, file access, 
etc.

Ideally these would all be served by a single, easilly extensible agent that 
can be deployed in a wide range of guests. Virtagent is an XMLRPC server, 
integrated into QEMU and a simple guest daemon, aimed at providing this type of 
functionality.

DESIGN:

There are actually 2 RPC servers:

1) a server in the guest agent which handles RPC requests from QEMU
2) a server in the host, integrated into the virtagent chardev, to handle RPC 
requests sent by the guest agent (mainly for handling asynchronous events 
reported by the agent).

Communication is done via RPCs (XMLRPC/HTTP between host and guest), albeit 
with a non-standard implementation that allows for multiplexing server/client 
RPC over a single virtio/isa serial channel.

EXAMPLE USAGE:

 - Configure guest agent to talk to host via virtio-serial
    # start guest with virtio-serial/virtagent. for example (RHEL6):
    qemu \
    -chardev virtagent,id=test0 \
    -device virtio-serial \
    -device virtserialport,chardev=test0,name=virtagent0 \
    -monitor stdio
    ...
    # in the guest:
    ./qemu-va -c virtio-serial -p /dev/virtio-ports/virtagent0
    ...
    # monitor commands
    (qemu) agent_viewdmesg
    [139311.710326] wlan0: deauthenticating from 00:30:bd:f7:12:d5 by local 
choice (reason=3)
    [139323.469857] wlan0: deauthenticating from 00:21:29:cd:41:ee by local 
choice (reason=3)
    ...
    [257683.375646] wlan0: authenticated
    [257683.375684] wlan0: associate with AP 00:30:bd:f7:12:d5 (try 1)
    [257683.377932] wlan0: RX AssocResp from 00:30:bd:f7:12:d5 (capab=0x411 
status=0 aid=4)
    [257683.377940] wlan0: associated
    
    (qemu) agent_viewfile /proc/meminfo
    MemTotal:        3985488 kB
    MemFree:          400524 kB
    Buffers:          220556 kB
    Cached:          2073160 kB
    SwapCached:            0 kB
    ...
    Hugepagesize:       2048 kB
    DirectMap4k:        8896 kB
    DirectMap2M:     4110336 kB
    (qemu) agent_shutdown powerdown
    (qemu)

KNOWN ISSUES/PLANS:
 - Implement RPC for guest script/command execution
 - Use UUIDs for generating unique tags for each http request/response
 - Scan for sentinel value while reading http content as well to immediately 
detect truncated requests/responses and avoid accidentally consuming new ones
 - switch to standard logging/trace mechanisms
 - outstanding requests should be reset if we get a hello notification (this 
implies guest/guest agent restart)
 - the client socket that qemu connects to send RPCs is a hardcoded filepath. 
This is unacceptable as the socket is channel/process specific and things will 
break when multiple guests are started.

[RFC][PATCH v6 01/23] Move code related to fd handlers into utility functions   
                                                                         
[RFC][PATCH v6 02/23] Add qemu_set_fd_handler() wrappers to qemu-tools.c
[RFC][PATCH v6 03/23] Make qemu timers available for tools
[RFC][PATCH v6 04/23] virtagent: common code for managing client/server rpc jobs
[RFC][PATCH v6 05/23] virtagent: transport definitions read/send callback 
functions
[RFC][PATCH v6 06/23] virtagent: base client definitions
[RFC][PATCH v6 07/23] virtagent: base server definitions
[RFC][PATCH v6 08/23] virtagent: add va.getfile RPC
[RFC][PATCH v6 09/23] virtagent: add agent_viewfile qmp/hmp command
[RFC][PATCH v6 10/23] virtagent: add va.getdmesg RPC
[RFC][PATCH v6 11/23] virtagent: add agent_viewdmesg qmp/hmp commands
[RFC][PATCH v6 12/23] virtagent: add va.shutdown RPC
[RFC][PATCH v6 13/23] virtagent: add agent_shutdown qmp/hmp commands
[RFC][PATCH v6 14/23] virtagent: add va.ping RPC
[RFC][PATCH v6 15/23] virtagent: add agent_ping qmp/hmp commands
[RFC][PATCH v6 16/23] virtagent: add agent_capabilities qmp/hmp commands
[RFC][PATCH v6 17/23] virtagent: add client capabilities init function
[RFC][PATCH v6 18/23] virtagent: add va.hello RPC
[RFC][PATCH v6 19/23] virtagent: add "hello" notification function for guest 
agent
[RFC][PATCH v6 20/23] virtagent: add va.capabilities RPC
[RFC][PATCH v6 21/23] virtagent: add virtagent guest daemon
[RFC][PATCH v6 22/23] virtagent: integrate virtagent server/client via chardev
[RFC][PATCH v6 23/23] virtagent: various bits to build QEMU with virtagent

 Makefile           |    4 +-
 Makefile.objs      |    2 +-
 Makefile.target    |    2 +-
 configure          |   32 ++
 cpus.c             |   83 +----
 hmp-commands.hx    |   80 ++++
 monitor.c          |    1 +
 qemu-char.c        |   44 +++
 qemu-char.h        |    4 +
 qemu-ioh.c         |  208 +++++++++++
 qemu-ioh.h         |   43 +++
 qemu-tool.c        |  115 ++++++-
 qemu-tool.h        |   26 ++
 qemu-va.c          |  238 ++++++++++++
 qerror.c           |    8 +
 qerror.h           |    6 +
 qmp-commands.hx    |  164 +++++++++
 virtagent-common.c | 1028 ++++++++++++++++++++++++++++++++++++++++++++++++++++
 virtagent-common.h |   73 ++++
 virtagent-server.c |  350 ++++++++++++++++++
 virtagent-server.h |   34 ++
 virtagent.c        |  642 ++++++++++++++++++++++++++++++++
 virtagent.h        |   50 +++
 vl.c               |   86 +----
 25 files changed, 3177 insertions(+), 148 deletions(-)