Re: [Qemu-devel] [RFC 0/2] virtio-vhost-user: add virtio-vhost-user device

[Jason Wang]

On 2018年01月23日 04:04, Michael S. Tsirkin wrote:
> On Mon, Jan 22, 2018 at 12:17:51PM +0000, Stefan Hajnoczi wrote:
> > On Mon, Jan 22, 2018 at 11:33:46AM +0800, Jason Wang wrote:
> > > On 2018年01月19日 21:06, Stefan Hajnoczi wrote:
> > > > These patches implement the virtio-vhost-user device design that I have
> > > > described here:
> > > > https://stefanha.github.io/virtio/vhost-user-slave.html#x1-2830007
> > > Thanks for the patches, looks rather interesting and similar to split device
> > > model used by Xen.
> > >
> > > > The goal is to let the guest act as the vhost device backend for other guests.
> > > > This allows virtual networking and storage appliances to run inside guests.
> > > So the question still, what kind of protocol do you want to run on top? If
> > > it was ethernet based, virtio-net work pretty well and it can even do
> > > migration.
> > >
> > > > This device is particularly interesting for poll mode drivers where exitless
> > > > VM-to-VM communication is possible, completely bypassing the hypervisor in the
> > > > data path.
> > > It's better to clarify the reason of hypervisor bypassing. (performance,
> > > security or scalability).
> > Performance - yes, definitely.  Exitless VM-to-VM is the fastest
> > possible way to communicate between VMs.  Today it can only be done
> > using ivshmem.  This patch series allows virtio devices to take
> > advantage of it and will encourage people to use virtio instead of
> > non-standard ivshmem devices.
> >
> > Security - I don't think this feature is a security improvement.  It
> > reduces isolation because VM1 has full shared memory access to VM2.  In
> > fact, this is a reason for users to consider carefully whether they
> > even want to use this feature.
> True without an IOMMU, however using a vIOMMU within VM2
> can protect the VM2, can't it?

It's not clear to me how to do this. E.g need a way to report failure to 
VM2 or #PF?

> > Scalability - much for the same reasons as the Performance section
> > above.  Bypassing the hypervisor eliminates scalability bottlenecks
> > (e.g. host network stack and bridge).
> >
> > > Probably not for the following cases:
> > >
> > > 1) kick/call
> > I disagree here because kick/call is actually very efficient!
> >
> > VM1's irqfd is the ioeventfd for VM2.  When VM2 writes to the ioeventfd
> > there is a single lightweight vmexit which injects an interrupt into
> > VM1.  QEMU is not involved and the host kernel scheduler is not involved
> > so this is a low-latency operation.

Right, looks like I was wrong. But consider irqfd may do wakup which 
means scheduler is still needed.

> > I haven't tested this yet but the ioeventfd code looks like this will
> > work.
> >
> > > 2) device IOTLB / IOMMU transaction (or any other case that backends needs
> > > metadata from qemu).
> > Yes, this is the big weakness of vhost-user in general.  The IOMMU
> > feature doesn't offer good isolation
> I think that's an implementation issue, not a protocol issue.
>
>
> > and even when it does, performance
> > will be an issue.
> If the IOMMU mappings are dynamic - but they are mostly
> static with e.g. dpdk, right?
>
>
> > > >    * Implement "Additional Device Resources over PCI" for shared memory,
> > > >      doorbells, and notifications instead of hardcoding a BAR with magic
> > > >      offsets into virtio-vhost-user:
> > > >      https://stefanha.github.io/virtio/vhost-user-slave.html#x1-2920007
> > > Does this mean we need to standardize vhost-user protocol first?
> > Currently the draft spec says:
> >
> >    This section relies on definitions from the Vhost-user Protocol [1].
> >
> >    [1] 
> > https://git.qemu.org/?p=qemu.git;a=blob_plain;f=docs/interop/vhost-user.txt;hb=HEAD
> >
> > Michael: Is it okay to simply include this link?
>
> It is OK to include normative and non-normative references,
> they go in the introduction and then you refer to them
> anywhere in the document.
>
>
> I'm still reviewing the draft.  At some level, this is a general tunnel
> feature, it can tunnel any protocol. That would be one way to
> isolate it.

Right, but it should not be the main motivation, consider we can tunnel 
any protocol on top of ethernet too.

> > > >    * Implement the VRING_KICK eventfd - currently vhost-user slaves must be poll
> > > >      mode drivers.
> > > >    * Optimize VRING_CALL doorbell with ioeventfd to avoid QEMU exit.
> > > The performance implication needs to be measured. It looks to me both kick
> > > and call will introduce more latency form the point of guest.
> > I described the irqfd + ioeventfd approach above.  It should be faster
> > than virtio-net + bridge today.
> >
> > > >    * vhost-user log feature
> > > >    * UUID config field for stable device identification regardless of PCI
> > > >      bus addresses.
> > > >    * vhost-user IOMMU and SLAVE_REQ_FD feature
> > > So an assumption is the VM that implements vhost backends should be at least
> > > as secure as vhost-user backend process on host. Could we have this
> > > conclusion?
> > Yes.
> >
> > Sadly the vhost-user IOMMU protocol feature does not provide isolation.
> > At the moment IOMMU is basically a layer of indirection (mapping) but
> > the vhost-user backend process still has full access to guest RAM :(.
> An important feature would be to do the isolation in the qemu.
> So trust the qemu running VM2 but not VM2 itself.

Agree, we'd better not consider VM is as secure as qemu.

> > > Btw, it's better to have some early numbers, e.g what testpmd reports during
> > > forwarding.
> > I need to rely on others to do this (and many other things!) because
> > virtio-vhost-user isn't the focus of my work.
> >
> > These patches were written to demonstrate my suggestions for vhost-pci.
> > They were written at work but also on weekends, early mornings, and late
> > nights to avoid delaying Wei and Zhiyong's vhost-pci work too much.

Thanks a lot for the effort! If anyone want to benchmark, I would expect 
compare the following three solutions:

1) vhost-pci
2) virtio-vhost-user
3) testpmd with two vhost-user ports


Performance number is really important to show the advantages of new ideas.

> > If this approach has merit then I hope others will take over and I'll
> > play a smaller role addressing some of the todo items and cleanups.

It looks to me the advantages are 1) generic virtio layer (vhost-pci can 
achieve this too if necessary) 2) some kind of code reusing (vhost pmd). 
And I'd expect they have similar performance result consider no major 
differences between them.

Thanks

> > Stefan