Re: [Qemu-devel] RFC: Universal encryption on QEMU I/O channels

[Dr. David Alan Gilbert]

* Daniel P. Berrange (address@hidden) wrote:
> In QEMU there are a number of features which involve communication with an
> external system over an I/O channel of some form. The features include
> migration, NBD, VNC and character devices. The I/O channel in question might
> might be a FIFO pipe, a PTY, a TCP socket, a UNIX domain socket, RMDA channel
> or something else, while the external system can be another QEMU, libvirt, an
> NBD server, or something else.

> Currently the only place where there is any meaningful level of security is
> the VNC server, which supports the VeNCrypt extension providing TLS sessions
> for data encryption and x509 cert validation for authentication, and the SASL
> extension which also provides either encryption of authentication or both.
> The migration data channel is more or less completely unprotected unless it
> is tunnelled via libvirt or equivalent external secure channel. The same is
> true for NBD, though there was a recent discussion about defining an extension
> to use TLS. Likewise serial ports, parallel ports, virtio consoles all use the
> chardev backends which offer no security features.

I think fixing this for migration might simplify stuff for users a lot as well;
the choice of whether libvirt does tunneling or not and what it means
for how block migration happens etc can get very confusing.

> There are some other network related pieces in QEMU, namely the built-in 
> iSCSI,
> RBD, NFS clients, and SPICE server. Since those are all implemented via 3rd
> party libraries rather than natively by QEMU I'm going ignore them for the 
> sake
> of this discussion and focus on network interaction directly implemented in
> QEMU.
> 
> We have a broad goal in OpenStack that every network channel in use must have
> encryption and authentication capabilities. Currently all the communication
> channels between the end user and the cloud infrastructure edge servers are
> secured, but internally a number of the cloud infrastructure components are
> unsecured. For example, we recommend to tunnel migration via libvirt, though
> that excludes use of the NBD for block migration since libvirt can't currently
> tunnel that. Internally we will shortly use  VeNCrypt for protecting VNC
> between the compute hosts and the openstack console proxy edge servers. We
> are lacking the abilty to secure serial ports between the compute nods and
> console proxy.
> 
> Essentially the project considers that it is no longer sufficient to consider
> the private management LAN (on which the cloud infrastructure is deployed) to
> be fully trusted; it must be considered hostile.
> 
> So back to QEMU/KVM, we want to have
> 
>  - Native support for encryption & authentication with migration, since
>    tunnelling via libvirt has a non-negligble performance overhead adding
>    both latency and CPU load
> 
>  - Native support for encryption & authentication with NBD server to enable
>    security of the block migration service
> 
>  - Native support for encryption & authentication with chardev TCP backends
>    to enable security of the serial port consoles.
> 
> As a starting point, the desire is to have TLS for session encryption and
> x509 certificate verification for authentication, but at a later date we'd
> also like to add the ability to use SASL for either aspect too. Thinking about
> QEMU users in general, it would probably be useful if any impl could allow for
> future enhancements such as SSH tunnelling too.
> 
> I have some development cycles available to work on addressing these gaps in
> QEMU, along with relevant experiance since I did the previous VNC server work
> for adding TLS and SASL in QEMU, along with similar in libvirt and GTK-VNC.
> 
> Having looked at the QEMU code for VNC, migration and chardevs I think the
> main stumbling block is currently that QEMU does not have any kind of standard
> approach for dealing with I/O channels internally. Migration has the QEMUFile
> abstraction, but it is currently fairly coupled to the migration code itself
> and limited in scope, because it doesn't actually deal with socket listen
> or connect tasks. That's still part of the migration code itself, QEMUFile
> only deals with I/O transfer, so it is a pretty incomplete abstraction layer.
> The chardev code has a backend abstraction, but that is really horribly
> entwined with the chardev code so not reusable in any way without a rewrite
> from scratch IMHO. The VNC server has alot of useful code for dealing with
> TLS & SASL, but it is somewhat entwined with the VNC server. The VNC server
> doesn't use any I/O abstraction just preferring raw FDs / sockets. I've not
> looked at the NBD code in detail, but I'm presuming it is using raw FDs /
> sockets.
> 
> Since this TLS/SASL code is non-trivial (and obviously security critical), I
> really don't want to end up with 4 separate places to implement it in QEMU.
> IMHO the only practical / sensible approach is to define some kind of standard
> I/O channel API internally to QEMU which migration, NBD, chardev and VNC all
> use. That gives us a single place to integrate all the security mechanisms
> we need to support.  In libvirt we did something like this a little while
> ago by defining a standard internal sockets API[1], with plugins for things
> like SASL[2], TLS[4] and SSH[5] and it has been very successful in simplifying
> the code by centralizing the hairy logic, though I wouldn't aim for exactly
> the same design if doing it again - a more layered approach like QEMU blockdev
> drivers is probably better in retrospect.
> 
> So my idea would be that we define a QEMUChannel object and set of APIs to
> standardize all interaction with sockets, pipes, RDMA, whatever $channel,

I'm not sure if it makes sense for RDMA; it already has a couple of hooks
that go around the back of QEMUFile in migration, and it's transferring the
data via DMA so the page data doesn't go through the same path.

> and then convert the QEMU features I've mentioned over to use that. I think
> that would be simpler than trying to untangle QEMUFile code from migration
> and then extend its features.
> 
> A rough plan of attack would be to split the work in a number of distinct
> pieces
> 
>  - Define a basic QEMUChannel object & APIs.
> 
>  - Convert chardev backend to the QEMUChannel API
> 
>  - Convert migration to the QEMUChannel API
> 
>  - Convert NBD to the QEMUChannel API
> 
>  - Introduce support for TLS to the QEMUChannel object
> 
>  - Introduce support for SASL to the QEMUChannel object
> 
>  - Convert VNC server to the QEMUChannel API
> 
>  - Integrate TLS in migration  (eg flags turn it on/off via CLI/monitor)
> 
>  - Integrate TLS in NBD (eg flags to turn it on/off via CLI/monitor)
> 
>  - Integrate TLS in chardev backend  (eg flags to turn it on/off via 
> CLI/monitor)
> 
> Aside from the first task, I'm not sure that's the best/right order to do
> the work, it is merely the rough set of pieces I see as required. I've also
> no idea how long this would take. It is clearly a pretty large task given
> the different areas of code it touchs.
> 
> In terms of openstack priorities though, migration is the most important,
> which would be tackling migration + NBD, with chardevs being a secondary
> importance. Converting VNC is obviously lowest priority since that already
> has security - it is just a sanity thing to avoid having two inmpls of TLS
> in QEMU long term. There would also need to be unit tests for key peices
> of functionality added along the way, especially for security critical tasks
> like TLS certificate validation.
> 
> It should go without saying, but just in case, any conversion of chardev
> and migration code would have to meet feature parity and especially
> performance parity for migration when compared like-for-like features.
> eg we should not regress in performance of uncrypted migration vs the
> current impl, but encrypted migration might be slower for obvious
> reasons (depends on level of encryption alg offload to the CPU)

Some things to keep in mind:
  1) The current patch from Liang Li doing multi threaded compression
     It just strikes me as an exmaple of another type of filter in the migration
     stream.

  2) Postcopy and fault tolerance need a bidirectional channel; and that back
     channel tends to be small messages.
  3) I'd considered making a separate socket/fd for passing page data
     in the hope of maybe making that page take data via splice; but am not
     sure yet.

Thanks for looking at this,

Dave

> Regards,
> Daniel
> 
> [1] http://libvirt.org/git/?p=libvirt.git;a=blob;f=src/rpc/virnetsocket.h
> [2] http://libvirt.org/git/?p=libvirt.git;a=blob;f=src/rpc/virnetsaslcontext.h
> [3] http://libvirt.org/git/?p=libvirt.git;a=blob;f=src/rpc/virnettlscontext.h
> [4] http://libvirt.org/git/?p=libvirt.git;a=blob;f=src/rpc/virnetsshsession.h
> -- 
> |: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
> |: http://libvirt.org              -o-             http://virt-manager.org :|
> |: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
> |: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|
> 
--
Dr. David Alan Gilbert / address@hidden / Manchester, UK