Re: [Qemu-devel] [RFC 0/2] virtio-vhost-user: add virtio-vhost-user device

[Michael S. Tsirkin]

On Mon, Jan 22, 2018 at 12:17:51PM +0000, Stefan Hajnoczi wrote:
> On Mon, Jan 22, 2018 at 11:33:46AM +0800, Jason Wang wrote:
> > On 2018年01月19日 21:06, Stefan Hajnoczi wrote:
> > > These patches implement the virtio-vhost-user device design that I have
> > > described here:
> > > https://stefanha.github.io/virtio/vhost-user-slave.html#x1-2830007
> > 
> > Thanks for the patches, looks rather interesting and similar to split device
> > model used by Xen.
> > 
> > > 
> > > The goal is to let the guest act as the vhost device backend for other 
> > > guests.
> > > This allows virtual networking and storage appliances to run inside 
> > > guests.
> > 
> > So the question still, what kind of protocol do you want to run on top? If
> > it was ethernet based, virtio-net work pretty well and it can even do
> > migration.
> > 
> > > This device is particularly interesting for poll mode drivers where 
> > > exitless
> > > VM-to-VM communication is possible, completely bypassing the hypervisor 
> > > in the
> > > data path.
> > 
> > It's better to clarify the reason of hypervisor bypassing. (performance,
> > security or scalability).
> 
> Performance - yes, definitely.  Exitless VM-to-VM is the fastest
> possible way to communicate between VMs.  Today it can only be done
> using ivshmem.  This patch series allows virtio devices to take
> advantage of it and will encourage people to use virtio instead of
> non-standard ivshmem devices.
> 
> Security - I don't think this feature is a security improvement.  It
> reduces isolation because VM1 has full shared memory access to VM2.  In
> fact, this is a reason for users to consider carefully whether they
> even want to use this feature.

True without an IOMMU, however using a vIOMMU within VM2
can protect the VM2, can't it?

> Scalability - much for the same reasons as the Performance section
> above.  Bypassing the hypervisor eliminates scalability bottlenecks
> (e.g. host network stack and bridge).
> 
> > Probably not for the following cases:
> > 
> > 1) kick/call
> 
> I disagree here because kick/call is actually very efficient!
> 
> VM1's irqfd is the ioeventfd for VM2.  When VM2 writes to the ioeventfd
> there is a single lightweight vmexit which injects an interrupt into
> VM1.  QEMU is not involved and the host kernel scheduler is not involved
> so this is a low-latency operation.
> 
> I haven't tested this yet but the ioeventfd code looks like this will
> work.
> 
> > 2) device IOTLB / IOMMU transaction (or any other case that backends needs
> > metadata from qemu).
> 
> Yes, this is the big weakness of vhost-user in general.  The IOMMU
> feature doesn't offer good isolation

I think that's an implementation issue, not a protocol issue.


> and even when it does, performance
> will be an issue.

If the IOMMU mappings are dynamic - but they are mostly
static with e.g. dpdk, right?


> > >   * Implement "Additional Device Resources over PCI" for shared memory,
> > >     doorbells, and notifications instead of hardcoding a BAR with magic
> > >     offsets into virtio-vhost-user:
> > >     https://stefanha.github.io/virtio/vhost-user-slave.html#x1-2920007
> > 
> > Does this mean we need to standardize vhost-user protocol first?
> 
> Currently the draft spec says:
> 
>   This section relies on definitions from the Vhost-user Protocol [1].
> 
>   [1] 
> https://git.qemu.org/?p=qemu.git;a=blob_plain;f=docs/interop/vhost-user.txt;hb=HEAD
> 
> Michael: Is it okay to simply include this link?


It is OK to include normative and non-normative references,
they go in the introduction and then you refer to them
anywhere in the document.


I'm still reviewing the draft.  At some level, this is a general tunnel
feature, it can tunnel any protocol. That would be one way to
isolate it.

> > >   * Implement the VRING_KICK eventfd - currently vhost-user slaves must 
> > > be poll
> > >     mode drivers.
> > >   * Optimize VRING_CALL doorbell with ioeventfd to avoid QEMU exit.
> > 
> > The performance implication needs to be measured. It looks to me both kick
> > and call will introduce more latency form the point of guest.
> 
> I described the irqfd + ioeventfd approach above.  It should be faster
> than virtio-net + bridge today.
> 
> > >   * vhost-user log feature
> > >   * UUID config field for stable device identification regardless of PCI
> > >     bus addresses.
> > >   * vhost-user IOMMU and SLAVE_REQ_FD feature
> > 
> > So an assumption is the VM that implements vhost backends should be at least
> > as secure as vhost-user backend process on host. Could we have this
> > conclusion?
> 
> Yes.
> 
> Sadly the vhost-user IOMMU protocol feature does not provide isolation.
> At the moment IOMMU is basically a layer of indirection (mapping) but
> the vhost-user backend process still has full access to guest RAM :(.

An important feature would be to do the isolation in the qemu.
So trust the qemu running VM2 but not VM2 itself.


> > Btw, it's better to have some early numbers, e.g what testpmd reports during
> > forwarding.
> 
> I need to rely on others to do this (and many other things!) because
> virtio-vhost-user isn't the focus of my work.
> 
> These patches were written to demonstrate my suggestions for vhost-pci.
> They were written at work but also on weekends, early mornings, and late
> nights to avoid delaying Wei and Zhiyong's vhost-pci work too much.
> 
> If this approach has merit then I hope others will take over and I'll
> play a smaller role addressing some of the todo items and cleanups.
> 
> Stefan