Re: [Qemu-devel] vhost-pci and virtio-vhost-user

[Jason Wang]

On 2018年01月11日 23:23, Stefan Hajnoczi wrote:
> On Thu, Jan 11, 2018 at 06:57:03PM +0800, Jason Wang wrote:
> > On 2018年01月11日 00:14, Stefan Hajnoczi wrote:
> > > Hi Wei,
> > > I wanted to summarize the differences between the vhost-pci and
> > > virtio-vhost-user approaches because previous discussions may have been
> > > confusing.
> > >
> > > vhost-pci defines a new virtio device type for each vhost device type
> > > (net, scsi, blk).  It therefore requires a virtio device driver for each
> > > device type inside the slave VM.
> > >
> > > Adding a new device type requires:
> > > 1. Defining a new virtio device type in the VIRTIO specification.
> > > 3. Implementing a new QEMU device model.
> > > 2. Implementing a new virtio driver.
> > >
> > > virtio-vhost-user is a single virtio device that acts as a vhost-user
> > > protocol transport for any vhost device type.  It requires one virtio
> > > driver inside the slave VM and device types are implemented using
> > > existing vhost-user slave libraries (librte_vhost in DPDK and
> > > libvhost-user in QEMU).
> > >
> > > Adding a new device type to virtio-vhost-user involves:
> > > 1. Adding any new vhost-user protocol messages to the QEMU
> > >      virtio-vhost-user device model.
> > > 2. Adding any new vhost-user protocol messages to the vhost-user slave
> > >      library.
> > > 3. Implementing the new device slave.
> > >
> > > The simplest case is when no new vhost-user protocol messages are
> > > required for the new device.  Then all that's needed for
> > > virtio-vhost-user is a device slave implementation (#3).  That slave
> > > implementation will also work with AF_UNIX because the vhost-user slave
> > > library hides the transport (AF_UNIX vs virtio-vhost-user).  Even
> > > better, if another person has already implemented that device slave to
> > > use with AF_UNIX then no new code is needed for virtio-vhost-user
> > > support at all!
> > >
> > > If you compare this to vhost-pci, it would be necessary to design a new
> > > virtio device, implement it in QEMU, and implement the virtio driver.
> > > Much of virtio driver is more or less the same thing as the vhost-user
> > > device slave but it cannot be reused because the vhost-user protocol
> > > isn't being used by the virtio device.  The result is a lot of
> > > duplication in DPDK and other codebases that implement vhost-user
> > > slaves.
> > >
> > > The way that vhost-pci is designed means that anyone wishing to support
> > > a new device type has to become a virtio device designer.  They need to
> > > map vhost-user protocol concepts to a new virtio device type.  This will
> > > be time-consuming for everyone involved (e.g. the developer, the VIRTIO
> > > community, etc).
> > >
> > > The virtio-vhost-user approach stays at the vhost-user protocol level as
> > > much as possible.  This way there are fewer concepts that need to be
> > > mapped by people adding new device types.  As a result, it will allow
> > > virtio-vhost-user to keep up with AF_UNIX vhost-user and grow because
> > > it's easier to work with.
> > >
> > > What do you think?
> > >
> > > Stefan
> > So a question is what's the motivation here?
> >
> > Form what I'm understanding, vhost-pci tries to build a scalable V2V private
> > datapath. But according to what you describe here, virito-vhost-user tries
> > to make it possible to implement the device inside another VM. I understand
> > the goal of vhost-pci could be done on top, but it looks to me it would then
> > rather similar to the design of Xen driver domain. So I can not figure out
> > how it can be done in a high performance way.
> vhost-pci and virtio-vhost-user both have the same goal.  They allow
> a VM to implement a vhost device (net, scsi, blk, etc).

Looks not, if I read the code correctly, vhost-pci has a device 
implementation in qemu, and in slave VM it only have a vhost-pci-net driver.

> This allows
> software defined network or storage appliances running inside a VM to
> provide I/O services to other VMs.

Well, I think we can do it even with the existed virtio or whatever 
other emulated device which should not be bounded to any specific kind 
of device.  And what's more important, according to the kvm 2016 slides 
of vhost-pci, the motivation of vhost-pci is not building SDN but a 
chain of VNFs. So bypassing the central vswitch through a private VM2VM 
path does make sense. (Though whether or not vhost-pci is the best 
choice is still questionable).

>   To the other VMs the devices look
> like regular virtio devices.
>
> I'm not sure I understand your reference to the Xen driver domain or
> performance.

So what proposed here is basically memory sharing and event notification 
through eventfd, this model have been used by Xen for many years through 
grant table and event channel. Xen use this to move the backend 
implementation from dom0 to a driver domain which has direct access to 
some hardwares. Consider the case of network, it can then implement xen 
netback inside driver domain which can access hardware NIC directly.

This makes sense for Xen and for performance since driver domain 
(backend) can access hardware directly and event was triggered through 
lower overhead hypercall (or it can do busypolling). But for 
virtio-vhost-user, unless you want SRIOV based solutions inside the 
slave VM, I believe we won't want to go back to Xen since the hardware 
virtualization can bring extra overheads.

>   Both vhost-pci and virtio-vhost-user work using shared
> memory access to the guest RAM of the other VM.  Therefore they can poll
> virtqueues and avoid vmexit.  They do also support cross-VM interrupts,
> thanks to QEMU setting up irqfd/ioeventfd appropriately on the host.
>
> Stefan

So in conclusion, consider the complexity, I would suggest to figure out 
whether or not this (either vhost-pci or virito-vhost-user) is really 
required before moving ahead. E.g, for VM2VM direct network path, this 
looks simply an issue of network topology instead of the problem of 
device, so there's a lot of trick, for vhost-user one can easily image 
to write an application (or use testpmd) to build a zerocopied VM2VM 
datapath, isn't this not sufficient for the case?

Thanks