Re: [Qemu-devel] vhost-pci and virtio-vhost-user

[Yang, Zhiyong]

Hi Jason,

> -----Original Message-----
> From: Qemu-devel [mailto:qemu-devel-
> address@hidden On Behalf Of Jason Wang
> Sent: Friday, January 12, 2018 11:33 AM
> To: Stefan Hajnoczi <address@hidden>
> Cc: Wang, Wei W <address@hidden>; address@hidden
> Subject: Re: [Qemu-devel] vhost-pci and virtio-vhost-user
> 
> 
> 
> On 2018年01月11日 23:23, Stefan Hajnoczi wrote:
> > On Thu, Jan 11, 2018 at 06:57:03PM +0800, Jason Wang wrote:
> >>
> >> On 2018年01月11日 00:14, Stefan Hajnoczi wrote:
> >>> Hi Wei,
> >>> I wanted to summarize the differences between the vhost-pci and
> >>> virtio-vhost-user approaches because previous discussions may have
> >>> been confusing.
> >>>
> >>> vhost-pci defines a new virtio device type for each vhost device
> >>> type (net, scsi, blk).  It therefore requires a virtio device driver
> >>> for each device type inside the slave VM.
> >>>
> >>> Adding a new device type requires:
> >>> 1. Defining a new virtio device type in the VIRTIO specification.
> >>> 3. Implementing a new QEMU device model.
> >>> 2. Implementing a new virtio driver.
> >>>
> >>> virtio-vhost-user is a single virtio device that acts as a
> >>> vhost-user protocol transport for any vhost device type.  It
> >>> requires one virtio driver inside the slave VM and device types are
> >>> implemented using existing vhost-user slave libraries (librte_vhost
> >>> in DPDK and libvhost-user in QEMU).
> >>>
> >>> Adding a new device type to virtio-vhost-user involves:
> >>> 1. Adding any new vhost-user protocol messages to the QEMU
> >>>      virtio-vhost-user device model.
> >>> 2. Adding any new vhost-user protocol messages to the vhost-user slave
> >>>      library.
> >>> 3. Implementing the new device slave.
> >>>
> >>> The simplest case is when no new vhost-user protocol messages are
> >>> required for the new device.  Then all that's needed for
> >>> virtio-vhost-user is a device slave implementation (#3).  That slave
> >>> implementation will also work with AF_UNIX because the vhost-user
> >>> slave library hides the transport (AF_UNIX vs virtio-vhost-user).
> >>> Even better, if another person has already implemented that device
> >>> slave to use with AF_UNIX then no new code is needed for
> >>> virtio-vhost-user support at all!
> >>>
> >>> If you compare this to vhost-pci, it would be necessary to design a
> >>> new virtio device, implement it in QEMU, and implement the virtio driver.
> >>> Much of virtio driver is more or less the same thing as the
> >>> vhost-user device slave but it cannot be reused because the
> >>> vhost-user protocol isn't being used by the virtio device.  The
> >>> result is a lot of duplication in DPDK and other codebases that
> >>> implement vhost-user slaves.
> >>>
> >>> The way that vhost-pci is designed means that anyone wishing to
> >>> support a new device type has to become a virtio device designer.
> >>> They need to map vhost-user protocol concepts to a new virtio device
> >>> type.  This will be time-consuming for everyone involved (e.g. the
> >>> developer, the VIRTIO community, etc).
> >>>
> >>> The virtio-vhost-user approach stays at the vhost-user protocol
> >>> level as much as possible.  This way there are fewer concepts that
> >>> need to be mapped by people adding new device types.  As a result,
> >>> it will allow virtio-vhost-user to keep up with AF_UNIX vhost-user
> >>> and grow because it's easier to work with.
> >>>
> >>> What do you think?
> >>>
> >>> Stefan
> >> So a question is what's the motivation here?
> >>
> >> Form what I'm understanding, vhost-pci tries to build a scalable V2V
> >> private datapath. But according to what you describe here,
> >> virito-vhost-user tries to make it possible to implement the device
> >> inside another VM. I understand the goal of vhost-pci could be done
> >> on top, but it looks to me it would then rather similar to the design
> >> of Xen driver domain. So I can not figure out how it can be done in a high
> performance way.
> > vhost-pci and virtio-vhost-user both have the same goal.  They allow a
> > VM to implement a vhost device (net, scsi, blk, etc).
> 
> Looks not, if I read the code correctly, vhost-pci has a device implementation
> in qemu, and in slave VM it only have a vhost-pci-net driver.
> 
> > This allows
> > software defined network or storage appliances running inside a VM to
> > provide I/O services to other VMs.
> 
> Well, I think we can do it even with the existed virtio or whatever other
> emulated device which should not be bounded to any specific kind of
> device.  And what's more important, according to the kvm 2016 slides of
> vhost-pci, the motivation of vhost-pci is not building SDN but a chain of 
> VNFs.
> So bypassing the central vswitch through a private VM2VM path does make
> sense. (Though whether or not vhost-pci is the best choice is still
> questionable).
> 
> >   To the other VMs the devices look
> > like regular virtio devices.
> >
> > I'm not sure I understand your reference to the Xen driver domain or
> > performance.
> 
> So what proposed here is basically memory sharing and event notification
> through eventfd, this model have been used by Xen for many years through
> grant table and event channel. Xen use this to move the backend
> implementation from dom0 to a driver domain which has direct access to
> some hardwares. Consider the case of network, it can then implement xen
> netback inside driver domain which can access hardware NIC directly.
> 
> This makes sense for Xen and for performance since driver domain
> (backend) can access hardware directly and event was triggered through
> lower overhead hypercall (or it can do busypolling). But for 
> virtio-vhost-user,
> unless you want SRIOV based solutions inside the slave VM, I believe we
> won't want to go back to Xen since the hardware virtualization can bring
> extra overheads.
> 
> >   Both vhost-pci and virtio-vhost-user work using shared memory access
> > to the guest RAM of the other VM.  Therefore they can poll virtqueues
> > and avoid vmexit.  They do also support cross-VM interrupts, thanks to
> > QEMU setting up irqfd/ioeventfd appropriately on the host.
> >
> > Stefan
> 
> So in conclusion, consider the complexity, I would suggest to figure out
> whether or not this (either vhost-pci or virito-vhost-user) is really required
> before moving ahead. E.g, for VM2VM direct network path, this looks simply
> an issue of network topology instead of the problem of device, so there's a
> lot of trick, for vhost-user one can easily image to write an application (or 
> use
> testpmd) to build a zerocopied VM2VM datapath, isn't this not sufficient for
> the case?

As far as I know,  dequeue zero copied feature of vhost user PMD can't help 
improve throughput for small packest ,such as 64 bytes.
On the contrary, it causes perf drop.  The feature mainly helps large packets 
throughput.
   
Vhostpci can bring the following advantages compared to traditional 
solution(vhost/virtio PMD pairs)
1.  higher throughput for two VMs. ( Let us see the following  case,  if we use 
NIC passthrough way to two 2 VMs,  vhostpci RX or TX is handled  running 1 
single core in VM1,  virtio PMD  is similar on VM2, 
Only RX or TX is handled running on one single core.
for traditional solution,  except each virtio PMD is running inside each VM,  
at least one extra core is needed for vhost user RX and TX as an mediator. 
In this case, the bottleneck lies in the two vhost user ports running on one 
single core, which has double workload. 

2. Low latencies (have shorter data path than tradition soluton, doesn't need 
to pass host OS any more by vhost user)

3. reduce nearly 50% cores  because  OVS is not involved again if we apply 
vhostpci/virtio to VMs-chain case.

Thanks
Zhiyong

> 
> Thanks