[Qemu-devel] [PATCH v1 RFC 00/34] Generic support for TLS protocol & I/O channels

[Daniel P. Berrange]

The following series is a work-in-progress of my effort to bring TLS
encryption support to all the QEMU socket based backends. The current
patches just illustrate changes to the chardev backends, but future
patches will cover NBD client/server and the migration client/server
code. The original discussion was here:

https://lists.gnu.org/archive/html/qemu-devel/2015-02/msg00529.html

The bulk of the interesting work here has focused on pulling our the
TLS code from the VNC server and turning it into a self-contained,
reusable API. This has been accompanied with a general consolidation
of all cryptographic related code. So we have one place in the code
base for dealing with cryptographic hash functions (md5, sha, etc),
ciphers (DES, AES, etc), and protocols (TLS). The benefit of this
is that the rest of the code base doesn't have to be littered with
#ifdef HAVE_GNUTLS conditionals - all the integration code for gnutls
is in one place.

Building on that, the next step has been to define a generic I/O
channels API (inspired by GIOChannel, but improved to better suit
QEMU's requirements). This provides a higher level API for dealing
with POSIX sockets, and running protocols such as TLS, SASL, and
WebSockets over them. This again allowed the VNC code to be further
untangled, so it doesn't have to directly know about TLS or WebSockets
for the most part. This will allow us to add WebSockets support to
the chardev backends too at some point.

The patches series is sequenced into a number of logical groupings,
with a view to allowing the patches to be incrementally merged,
rather than having to take the entire series in one go.

 - QOM - a handful of minor enhancements & fixes to QOM, in
   particular some work to make handling of enum properties
   clearer, and to make instantiation of objects with properties
   simpler.

 - Crypto - introduce crypto/ and include/crypto/ directories
   that contain APIs for hash functions, cipher functions and
   the TLS credential (x509 cert) handling and TLS protocol
   itself. This will be further enhanced down the line as &
   when I get time to integrate support for LUKS.

 - Hash conversion - the quorum blockdrv and VNC server are
   converted to use the new crypto hash functions. This removes
   some of the #ifdef CONFIG_* conditionals from their codepaths

 - Cipher conversion - the qcow(2) blockdrv and VNC server are
   converted to use the new crypto cipher functions. This removes
   their direct dependency on QEMU's in-tree AES and DES impls. If
   QEMU is linked to GNUTLS, this will now transparently use either
   libgcrypt or nettle for the AES & DES algorithms. These impls are
   more actively maintained than QEMU's built-in impls and also
   benefit from FIPS certification in some distros. The built-in
   impls of course remain for those building without GNUTLS.

 - I/O Channels - introduce io/ and include/io/ directories that
   contain APIs for dealing with arbitrary bidirectional I/O channels.
   These can be POSIX sockets, fifos, pipes, or higher level layers
   such as TLS or WebSockets. Having a common API for all these
   concepts greatly facilitates the integration of TLS/WebSockets into
   the various areas of code in QEMU that are traditionally hardcoded
   to directly use the POSIX sockets API.

 - VNC conversion - convert the VNC server to use the new I/O channels
   APIs. This enables nearly all the remaining #ifdef CONFIG_*
   conditionals to be removed from the VNC codebase, greatly
   clarifying its code.

 - Chardev conversion - convert the chardev backend to use the new I/O
   channels APIs instead of GLib's GIOChannel. This also includes
   support for enabling TLS on the TCP chardev backend, nicely
   illustrating how the I/O channels APIs simplify support for such
   protocols.

I've attempted to get fairly complete API documentation coverage for
all the new code files I've created here, with example code usage too.
In addition, the crypto modules gain a number of new test suites to
validate correctness of the implementations, since this is critical
code to get right.

What I see still to be done (high == merge block, medium == nice to
have for merge, low == do it later):

 - The code is not fully ported to the Windows platform yet. In
   particular the QIOChannelSocket and QIOChannelFile classes are
   almost certainly broken & if they compile it is just luck. High
   priority.

 - Need to validate the reference counting / lifecycle of the
   QIOChannel objects in chardev & vnc servers to ensure I've not
   introduced any race conditions in their usage vs client disconnects.
   High priority.

 - Unit test coverage of all the QIOChannel subclasses to validate
   their correct operation. Medium priority

 - A QIOChannelMemory implementation that provides a memory buffer
   backed I/O channel. Mostly this will be used for the unit test
   suite, but might find other uses at some point. Medium priority.

 - APIs for establishing socket connections. Currently the
   QIOChannelSocket class is instantiated from a pre-connected socket
   file descriptor. It is desirable to have a constructor that just
   accepts a hostname/service/family and then performs the name
   resolution & connection code. This will make the API more consistent
   to use. Medium priority.

 - A QIOChannelSASL implementation that provides integration for the
   SASL authentication protocol. This will allow the last custom I/O
   layer to be removed from the VNC server code. Low priority, since
   we don't immediately want/need SASL support in chardev/migraton/nbd
   code.

 - A QIOChannelTelnet implementation that runs the telnet protocol, to
   replace the hacky telnet support that is hardwired into the chardev
   backends. Low priority.

 - The crypto API could usefully gain a cipher backend that uses the
   Linux kernel crypto API as an alternative to nettle or libgcrypt.
   Low priority, nice to have.

While I will probably start work on it, I'm not intending to submit
the update of the NBD/migration code, until this series has been
positively reviewed and looks like it is close to accepted for merge,
as there are already enough patches in this series as it is :-) The
aim though is to convert the NBD code to use QIOChannel instead of
direct sockets usage & add the TLS protocol extension previously
discussed with the NBD spec maintainer. The migration code will
either be adapted to use QIOChannel, or the QEMUFile code will be
adapted to use QIOChannel. Undecided which is the best approach there
at this time. Probably depends whether we can do a QIOChannelRDMA
impl that has performance on parity with what exists today.

For those interesting in testing I have made the series available
on github too

  https://github.com/berrange/qemu/tree/qemu-io-channel-7

The diffstat may look alarming but a good portion is in the test
suite and there's some quite verbose comments inline too which
bulk it up:

Daniel P. Berrange (34):
  ui: remove check for failure of qemu_acl_init()
  qom: document user creatable object types in help text
  qom: create objects in two phases
  qom: add object_new_propv / object_new_proplist constructors
  qom: make enum string tables const-correct
  qom: add a object_property_add_enum helper method
  qom: don't pass string table to object_get_enum method
  crypto: introduce new module for computing hash digests
  crypto: move built-in AES implementation into crypto/
  crypto: move built-in D3DES implementation into crypto/
  crypto: introduce generic cipher API & built-in implementation
  crypto: add a gcrypt cipher implementation
  crypto: add a nettle cipher implementation
  crypto: introduce new module for handling TLS credentials
  crypto: add sanity checking of TLS credentials
  crypto: introduce new module for handling TLS sessions
  block: convert quorum blockdrv to use crypto APIs
  ui: convert VNC websockets to use crypto APIs
  block: convert qcow/qcow2 to use generic cipher API
  ui: convert VNC to use generic cipher API
  io: add abstract QIOChannel classes
  io: add helper module for creating watches on UNIX FDs
  io: add QIOChannelSocket class
  io: add QIOChannelFile class
  io: add QIOTask class for async operations
  io: add QIOChannelTLS class
  io: pull Buffer code out of VNC module
  io: add QIOChannelWebsock class
  ui: convert VNC server to use QEMUIOChannelSocket classes
  ui: convert VNC server to use QIOChannelTLS
  ui: convert VNC server to use QIOChannelWebsock
  char: convert from GIOChannel to QIOChannel
  char: don't assume telnet initialization will not block
  char: introduce support for TLS encrypted TCP chardev backend

 Makefile.objs                         |    1 +
 backends/hostmem.c                    |   22 +-
 block/Makefile.objs                   |    2 +-
 block/qcow.c                          |  100 ++-
 block/qcow2-cluster.c                 |   46 +-
 block/qcow2.c                         |   95 +--
 block/qcow2.h                         |   13 +-
 block/quorum.c                        |   38 +-
 configure                             |  213 ++++---
 crypto/Makefile.objs                  |    7 +
 {util => crypto}/aes.c                |    2 +-
 crypto/cipher-builtin.c               |  391 ++++++++++++
 crypto/cipher-gcrypt.c                |  204 ++++++
 crypto/cipher-nettle.c                |  226 +++++++
 crypto/cipher.c                       |   31 +
 ui/d3des.c => crypto/desrfb.c         |    2 +-
 crypto/hash.c                         |  202 ++++++
 crypto/init.c                         |  160 +++++
 crypto/tlscreds.c                     | 1093 ++++++++++++++++++++++++++++++++
 crypto/tlssession.c                   |  546 ++++++++++++++++
 include/{qemu => crypto}/aes.h        |    0
 include/crypto/cipher.h               |  205 ++++++
 ui/d3des.h => include/crypto/desrfb.h |    0
 include/crypto/hash.h                 |  189 ++++++
 include/crypto/init.h                 |   29 +
 include/crypto/tlscreds.h             |  135 ++++
 include/crypto/tlssession.h           |  313 ++++++++++
 include/hw/qdev-core.h                |    2 +-
 include/io/buffer.h                   |  118 ++++
 include/io/channel-file.h             |   67 ++
 include/io/channel-socket.h           |  168 +++++
 include/io/channel-tls.h              |  142 +++++
 include/io/channel-unix.h             |   50 ++
 include/io/channel-websock.h          |  108 ++++
 include/io/channel.h                  |  388 ++++++++++++
 include/io/task.h                     |  168 +++++
 include/qapi/util.h                   |    2 +-
 include/qapi/visitor-impl.h           |    6 +-
 include/qapi/visitor.h                |    2 +-
 include/qom/object.h                  |   78 ++-
 io/Makefile.objs                      |    8 +
 io/buffer.c                           |   65 ++
 io/channel-file.c                     |  198 ++++++
 io/channel-socket.c                   |  572 +++++++++++++++++
 io/channel-tls.c                      |  393 ++++++++++++
 io/channel-unix.c                     |  100 +++
 io/channel-websock.c                  |  976 +++++++++++++++++++++++++++++
 io/channel.c                          |  178 ++++++
 io/task.c                             |   84 +++
 numa.c                                |    1 -
 qapi-schema.json                      |    2 +
 qapi/qapi-dealloc-visitor.c           |    3 +-
 qapi/qapi-util.c                      |    2 +-
 qapi/qapi-visit-core.c                |    6 +-
 qemu-char.c                           |  798 ++++++++++++------------
 qemu-options.hx                       |  137 +++-
 qom/object.c                          |  141 ++++-
 scripts/qapi-types.py                 |    4 +-
 target-arm/crypto_helper.c            |    2 +-
 target-i386/fpu_helper.c              |    1 -
 target-i386/ops_sse.h                 |    2 +-
 target-ppc/int_helper.c               |    2 +-
 tests/.gitignore                      |    9 +
 tests/Makefile                        |   16 +-
 tests/crypto-tls-helpers.c            |  485 +++++++++++++++
 tests/crypto-tls-helpers.h            |  133 ++++
 tests/pkix_asn1_tab.c                 | 1103 +++++++++++++++++++++++++++++++++
 tests/test-crypto-cipher.c            |  290 +++++++++
 tests/test-crypto-hash.c              |  209 +++++++
 tests/test-crypto-tlscreds.c          |  727 ++++++++++++++++++++++
 tests/test-crypto-tlssession.c        |  540 ++++++++++++++++
 ui/Makefile.objs                      |    6 +-
 ui/vnc-auth-sasl.c                    |   81 +--
 ui/vnc-auth-vencrypt.c                |   90 ++-
 ui/vnc-enc-tight.c                    |   38 +-
 ui/vnc-enc-zlib.c                     |    6 +-
 ui/vnc-enc-zrle.c                     |   18 +-
 ui/vnc-jobs.c                         |   25 +-
 ui/vnc-tls.c                          |   24 +-
 ui/vnc-tls.h                          |   69 ---
 ui/vnc-ws.c                           |  393 ++----------
 ui/vnc-ws.h                           |   75 +--
 ui/vnc.c                              |  997 ++++++++++++++---------------
 ui/vnc.h                              |  104 ++--
 util/Makefile.objs                    |    2 +-
 vl.c                                  |   37 +-
 86 files changed, 12868 insertions(+), 1848 deletions(-)
 create mode 100644 crypto/Makefile.objs
 rename {util => crypto}/aes.c (99%)
 create mode 100644 crypto/cipher-builtin.c
 create mode 100644 crypto/cipher-gcrypt.c
 create mode 100644 crypto/cipher-nettle.c
 create mode 100644 crypto/cipher.c
 rename ui/d3des.c => crypto/desrfb.c (99%)
 create mode 100644 crypto/hash.c
 create mode 100644 crypto/init.c
 create mode 100644 crypto/tlscreds.c
 create mode 100644 crypto/tlssession.c
 rename include/{qemu => crypto}/aes.h (100%)
 create mode 100644 include/crypto/cipher.h
 rename ui/d3des.h => include/crypto/desrfb.h (100%)
 create mode 100644 include/crypto/hash.h
 create mode 100644 include/crypto/init.h
 create mode 100644 include/crypto/tlscreds.h
 create mode 100644 include/crypto/tlssession.h
 create mode 100644 include/io/buffer.h
 create mode 100644 include/io/channel-file.h
 create mode 100644 include/io/channel-socket.h
 create mode 100644 include/io/channel-tls.h
 create mode 100644 include/io/channel-unix.h
 create mode 100644 include/io/channel-websock.h
 create mode 100644 include/io/channel.h
 create mode 100644 include/io/task.h
 create mode 100644 io/Makefile.objs
 create mode 100644 io/buffer.c
 create mode 100644 io/channel-file.c
 create mode 100644 io/channel-socket.c
 create mode 100644 io/channel-tls.c
 create mode 100644 io/channel-unix.c
 create mode 100644 io/channel-websock.c
 create mode 100644 io/channel.c
 create mode 100644 io/task.c
 create mode 100644 tests/crypto-tls-helpers.c
 create mode 100644 tests/crypto-tls-helpers.h
 create mode 100644 tests/pkix_asn1_tab.c
 create mode 100644 tests/test-crypto-cipher.c
 create mode 100644 tests/test-crypto-hash.c
 create mode 100644 tests/test-crypto-tlscreds.c
 create mode 100644 tests/test-crypto-tlssession.c
 delete mode 100644 ui/vnc-tls.h

-- 
2.1.0