Re: [Qemu-devel] [RFC v2 0/8] VIRTIO-IOMMU device

[Tian, Kevin]

> From: Jean-Philippe Brucker [mailto:address@hidden
> Sent: Wednesday, July 5, 2017 8:45 PM
> 
> On 05/07/17 08:14, Tian, Kevin wrote:
> >> From: Jean-Philippe Brucker [mailto:address@hidden
> >> Sent: Monday, June 19, 2017 6:15 PM
> >>
> >> On 19/06/17 08:54, Bharat Bhushan wrote:
> >>> Hi Eric,
> >>>
> >>> I started added replay in virtio-iommu and came across how MSI
> interrupts
> >> with work with VFIO.
> >>> I understand that on intel this works differently but vsmmu will have
> same
> >> requirement.
> >>> kvm-msi-irq-route are added using the msi-address to be translated by
> >> viommu and not the final translated address.
> >>> While currently the irqfd framework does not know about emulated
> >> iommus (virtio-iommu, vsmmuv3/vintel-iommu).
> >>> So in my view we have following options:
> >>> - Programming with translated address when setting up kvm-msi-irq-
> route
> >>> - Route the interrupts via QEMU, which is bad from performance
> >>> - vhost-virtio-iommu may solve the problem in long term
> >>>
> >>> Is there any other better option I am missing?
> >>
> >> Since we're on the topic of MSIs... I'm currently trying to figure out how
> >> we'll handle MSIs in the nested translation mode, where the guest
> manages
> >> S1 page tables and the host doesn't know about GVA->GPA translation.
> >>
> >> I'm also wondering about the benefits of having SW-mapped MSIs in the
> >> guest. It seems unavoidable for vSMMU since that's what a physical
> system
> >> would do. But in a paravirtualized solution there doesn't seem to be any
> >> compelling reason for having the guest map MSI doorbells. These
> addresses
> >> are never accessed directly, they are only used for setting up IRQ routing
> >> (at least on kvmtool). So here's what I'd like to have. Note that I
> >> haven't investigated the feasibility in Qemu yet, I don't know how it
> >> deals with MSIs.
> >>
> >> (1) Guest uses the guest-physical MSI doorbell when setting up MSIs. For
> >> ARM with GICv3 this would be GITS_TRANSLATER, for x86 it would be the
> >> fixed MSI doorbell. This way the host wouldn't need to inspect IOMMU
> >> mappings when handling writes to PCI MSI-X tables.
> >>
> >
> > What do you mean by "fixed MSI doorbell"? PCI MSI-X table is part of
> > PCI MMIO bar. Accessing to it is just a memory virtualization issue (e.g.
> > trap by KVM and then emulated in Qemu) on x86. It's not a IOMMU
> > problem. I guess you may mean same thing but want to double confirm
> > here given the terminology confusion. Or do you mean the interrupt
> > triggered by IOMMU itself?
> 
> Yes I didn't mean access to the MSI-X table, but how we interpret the
> address in the MSI message. In kvmtool I create MSI routes for VFIO
> devices when the guest accesses the MSI-X tables. And on ARM the tables
> contains an IOVA that needs to be translated into a PA, so handling a
> write to an MSI-X entry might mean doing the IOVA->PA translation of the
> doorbell.
> 
> On x86 the MSI address is 0xfee...., whether there is an IOMMU or not.
> That's what I meant by fixed. And it is the IOMMU that performs IRQ
> remapping.
> 
> On physical ARM systems, the SMMU doesn't treat any special address range
> as "MSI window". For the SMMU, an MSI is simply a memory transaction.
> MSI
> addresses are arbitrary IOVAs that get translated into PAs by the SMMU.
> The SMMU doesn't perform any IRQ remapping, only address translation.
> This
> PA is a doorbell register in the irqchip, which performs IRQ remapping and
> triggers an interrupt.

Thanks for explanation. I see the background now.

> 
> Therefore in an emulated ARM system, when the guest writes the MSI-X
> table, it writes an IOVA. In a strict emulation the MSI would have to
> first go through the vIOMMU, and then into the irqchip. I was wondering if
> with virtio-iommu we could skip the address translation and go to the MSI
> remapping component immediately, effectively implementing a "hardware
> MSI
> window". This is what x86 does, the difference being that MSI remapping is
> done by the IOMMU on x86, and by the irqchip on ARM.

sorry I didn't quite get this part, and here is my understanding:

Guest programs vIOMMU to map a gIOVA (used by MSI to a GPA 
of doorbell register of virtual irqchip. vIOMMU then 
triggers VFIO map/unmap to update physical IOMMU page
table for gIOVA -> HPA of real doorbell of physical irqchip
(assume your irqchip will provide multiple doorbells so each
device can have its own channel). then once this update is
done, later MSI interrupts from assigned device will go 
through physical IOMMU (gIOVA->HPA) then reach irqchip 
for irq remapping. vIOMMU is involved only in configuration
path instead of actual interrupt path.

If my understanding is correct, above will be the natural flow then
why is additional virtio-iommu change required? :-)

> 
> My current take is that we should keep the current behavior, but I will
> try to sort out the different ways of implementing MSIs with virtio-iommu
> in the next specification draft.
> 
> Thanks,
> Jean