[Qemu-devel] [PATCH V4 00/10] vhost device IOTLB support

[Jason Wang]

Hi all:

As the userspace vitio driver became popular, more and more request
were received for secure DMA environemt (DMAR). So this series tries
to make DMAR works for virtio/vhost. The idea is let virtio/vhost
co-work with userspace iommu implememtation. This is done through:

- for virtio, when platform supports IOMMU (VIRTIO_F_IOMMU_PLATFORM),
  virtio will not assume address_space_memory, instead a transport
  specific method were introduced for querying the dma address space
  and dma helpers were used in device emulation codes.
- for vhost, implement a device IOTLB by using device IOTLB API
  supported by recent kernel. With this API, vhost kernel can query
  IOTLB entry for a specified iova from qemu, qemu can invalidate an
  arbitrary range of iova in vhost kernel.

The device IOTLB API is totaly architecture independent, an example
implementation was done with intel iommu by:

- implement basic ATS (Address Translation Service) for virtio-pci,
  this will make device IOTLB visible for iommu driver in guest.
- implement device IOTLB descriptor processing in intel iommu (enabled
  through device-iotlb=on), and trigger the device IOTLB invalidation
  in vhost through iommu notifier.

It could be easily ported to other IOMMU or architecture even if it
doesn't support device IOTLB. (e.g just invalidate the vhost IOTLB
during IOMMU IOTLB invalidation). But this will be slow since several
devics were contending userspace IOTLB entries.

AMD IOMMU suppot for device IOTLB is ready, but it depends on other
fixes to work correctly.

Test was done by:

- intel_iommu=on/strict in guest.
- vfio l2fwd in guest.

This main use case is the programs that use fixed mapping in guest
(e.g dpdk). If 1G hugepage were used in guest, thanks to the SLLPS
support, we can get 100% TLB hit rate for l2fwd in guest.

For the normal kernel driver which uses lots of dynamic mapping and
unmapping, we may see performance penalty, this could be optimized in
the future.

TODO:
- more platforms and IOMMU support (done but block by other bugs of
  AMD IOMMU)
- performance optimizations (e.g merging adjacent mappings)
- non ATS support (userspace IOTLB snooping)
- using ATSCtl to disable Device IOTLB

Changes from V3:
- rebase to HEAD
- fill all entry of IOMMUTLBEntry before notify

Changes from V2:
- rebase to HEAD
- avoid querying dma_as each time by using vdev->dma_as directly
- fix and improve address_space_get_iotlb_entry()
- drop patch 1 (which has been posted as an independent fix)
- fix ECAP when device-iotlb=off
- fix centos6 build

Changes from V1:
- rebase to HEAD
- avoid calling transport specific dma as fetching method each time by
  caching it in vdev
- convert to use new IOMMU notifier API
- silent checkpatch warnings and fix 32bit build
- use "device-iotlb" instead of "device_iotlb"
- rename virtio_memory_map() to vhost_memory_map() and move it to vhost.c
- use memory_region_is_iommu() instead of inventing new one

Changes from RFC:
- rebase to HEAD
- switch to use new vhost device IOTLB API
- use the new feature bit VIRITO_F_IOMMU_PLATFORM
- finalize basic ATS implementation
- add ATSR for Root port ATS transaction
- fix the iommu notifier handling during unregistering
- use snprintf() in patch 3
- correc the loop in address_space_get_iotlb_entry()
- small tweak on the address calculation during device iotlb
  descriptor processing.

Jason Wang (10):
  virtio: convert to use DMA api
  intel_iommu: name vtd address space with devfn
  intel_iommu: allocate new key when creating new address space
  exec: introduce address_space_get_iotlb_entry()
  intel_iommu: support device iotlb descriptor
  virtio-pci: address space translation service (ATS) support
  acpi: add ATSR for q35
  memory: handle alias for iommu notifier
  memory: handle alias in memory_region_is_iommu()
  vhost_net: device IOTLB support

 exec.c                                    |  33 ++++++
 hw/block/virtio-blk.c                     |   2 +-
 hw/char/virtio-serial-bus.c               |   3 +-
 hw/i386/acpi-build.c                      |   9 ++
 hw/i386/intel_iommu.c                     |  92 +++++++++++++--
 hw/i386/intel_iommu_internal.h            |  13 ++-
 hw/i386/x86-iommu.c                       |  17 +++
 hw/pci/pcie.c                             |  16 +++
 hw/scsi/virtio-scsi.c                     |   4 +-
 hw/virtio/vhost-backend.c                 |  99 +++++++++++++++++
 hw/virtio/vhost.c                         | 178 ++++++++++++++++++++++++++----
 hw/virtio/virtio-bus.c                    |   8 ++
 hw/virtio/virtio-pci.c                    |  21 ++++
 hw/virtio/virtio-pci.h                    |   4 +
 hw/virtio/virtio.c                        |  57 ++++++----
 include/exec/memory.h                     |   8 ++
 include/hw/acpi/acpi-defs.h               |  12 ++
 include/hw/i386/x86-iommu.h               |   1 +
 include/hw/pci/pcie.h                     |   4 +
 include/hw/virtio/vhost-backend.h         |  13 +++
 include/hw/virtio/vhost.h                 |   4 +
 include/hw/virtio/virtio-access.h         |  31 ++++--
 include/hw/virtio/virtio-bus.h            |   1 +
 include/hw/virtio/virtio.h                |   9 +-
 include/standard-headers/linux/pci_regs.h |   1 +
 memory.c                                  |   9 ++
 net/tap.c                                 |   1 +
 27 files changed, 582 insertions(+), 68 deletions(-)

-- 
2.7.4