This RFC patch set does some small extensions to vhost-user protocol
to support VFIO based accelerators, and makes it possible to get the
similar performance of VFIO passthru while keeping the virtio device
emulation in QEMU.
When we have virtio ring compatible devices, it's possible to setup
the device (DMA mapping, PCI config, etc) based on the existing info
(memory-table, features, vring info, etc) which is available on the
vhost-backend (e.g. DPDK vhost library). Then, we will be able to
use such devices to accelerate the emulated device for the VM. And
we call it vDPA: vhost DataPath Acceleration. The key difference
between VFIO passthru and vDPA is that, in vDPA only the data path
(e.g. ring, notify and queue interrupt) is pass-throughed, the device
control path (e.g. PCI configuration space and MMIO regions) is still
defined and emulated by QEMU.
The benefits of keeping virtio device emulation in QEMU compared
with virtio device VFIO passthru include (but not limit to):
- consistent device interface from guest OS;
- max flexibility on control path and hardware design;
- leveraging the existing virtio live-migration framework;
But the critical issue in vDPA is that the data path performance is
relatively low and some host threads are needed for the data path,
because some necessary mechanisms are missing to support:
1) guest driver notifies the device directly;
2) device interrupts the guest directly;
So this patch set does some small extensions to vhost-user protocol
to make both of them possible. It leverages the same mechanisms (e.g.
EPT and Posted-Interrupt on Intel platform) as the VFIO passthru to
achieve the data path pass through.
A new protocol feature bit is added to negotiate the accelerator feature
support. Two new slave message types are added to enable the notify and
interrupt passthru for each queue. From the view of vhost-user protocol
design, it's very flexible. The passthru can be enabled/disabled for
each queue individually, and it's possible to accelerate each queue by
different devices. More design and implementation details can be found
from the last patch.
There are some rough edges in this patch set (so this is a RFC patch
set for now), but it's never too early to hear the thoughts from the
community! So any comments and suggestions would be really appreciated!
Tiwei Bie (3):
vhost-user: support receiving file descriptors in slave_read
vhost-user: introduce shared vhost-user state
vhost-user: add VFIO based accelerators support
docs/interop/vhost-user.txt | 57 ++++++
hw/scsi/vhost-user-scsi.c | 6 +-
hw/vfio/common.c | 2 +-
hw/virtio/vhost-user.c | 430
+++++++++++++++++++++++++++++++++++++++-
hw/virtio/vhost.c | 3 +-
hw/virtio/virtio-pci.c | 8 -
hw/virtio/virtio-pci.h | 8 +
include/hw/vfio/vfio.h | 2 +
include/hw/virtio/vhost-user.h | 43 ++++
include/hw/virtio/virtio-scsi.h | 6 +-
net/vhost-user.c | 30 +--
11 files changed, 561 insertions(+), 34 deletions(-)
create mode 100644 include/hw/virtio/vhost-user.h