Re: [Qemu-devel] [RFC v2 0/8] VIRTIO-IOMMU device

[Jean-Philippe Brucker]

On 05/07/17 08:14, Tian, Kevin wrote:
>> From: Jean-Philippe Brucker [mailto:address@hidden
>> Sent: Monday, June 19, 2017 6:15 PM
>>
>> On 19/06/17 08:54, Bharat Bhushan wrote:
>>> Hi Eric,
>>>
>>> I started added replay in virtio-iommu and came across how MSI interrupts
>> with work with VFIO.
>>> I understand that on intel this works differently but vsmmu will have same
>> requirement.
>>> kvm-msi-irq-route are added using the msi-address to be translated by
>> viommu and not the final translated address.
>>> While currently the irqfd framework does not know about emulated
>> iommus (virtio-iommu, vsmmuv3/vintel-iommu).
>>> So in my view we have following options:
>>> - Programming with translated address when setting up kvm-msi-irq-route
>>> - Route the interrupts via QEMU, which is bad from performance
>>> - vhost-virtio-iommu may solve the problem in long term
>>>
>>> Is there any other better option I am missing?
>>
>> Since we're on the topic of MSIs... I'm currently trying to figure out how
>> we'll handle MSIs in the nested translation mode, where the guest manages
>> S1 page tables and the host doesn't know about GVA->GPA translation.
>>
>> I'm also wondering about the benefits of having SW-mapped MSIs in the
>> guest. It seems unavoidable for vSMMU since that's what a physical system
>> would do. But in a paravirtualized solution there doesn't seem to be any
>> compelling reason for having the guest map MSI doorbells. These addresses
>> are never accessed directly, they are only used for setting up IRQ routing
>> (at least on kvmtool). So here's what I'd like to have. Note that I
>> haven't investigated the feasibility in Qemu yet, I don't know how it
>> deals with MSIs.
>>
>> (1) Guest uses the guest-physical MSI doorbell when setting up MSIs. For
>> ARM with GICv3 this would be GITS_TRANSLATER, for x86 it would be the
>> fixed MSI doorbell. This way the host wouldn't need to inspect IOMMU
>> mappings when handling writes to PCI MSI-X tables.
>>
> 
> What do you mean by "fixed MSI doorbell"? PCI MSI-X table is part of
> PCI MMIO bar. Accessing to it is just a memory virtualization issue (e.g.
> trap by KVM and then emulated in Qemu) on x86. It's not a IOMMU
> problem. I guess you may mean same thing but want to double confirm
> here given the terminology confusion. Or do you mean the interrupt
> triggered by IOMMU itself?

Yes I didn't mean access to the MSI-X table, but how we interpret the
address in the MSI message. In kvmtool I create MSI routes for VFIO
devices when the guest accesses the MSI-X tables. And on ARM the tables
contains an IOVA that needs to be translated into a PA, so handling a
write to an MSI-X entry might mean doing the IOVA->PA translation of the
doorbell.

On x86 the MSI address is 0xfee...., whether there is an IOMMU or not.
That's what I meant by fixed. And it is the IOMMU that performs IRQ remapping.

On physical ARM systems, the SMMU doesn't treat any special address range
as "MSI window". For the SMMU, an MSI is simply a memory transaction. MSI
addresses are arbitrary IOVAs that get translated into PAs by the SMMU.
The SMMU doesn't perform any IRQ remapping, only address translation. This
PA is a doorbell register in the irqchip, which performs IRQ remapping and
triggers an interrupt.

Therefore in an emulated ARM system, when the guest writes the MSI-X
table, it writes an IOVA. In a strict emulation the MSI would have to
first go through the vIOMMU, and then into the irqchip. I was wondering if
with virtio-iommu we could skip the address translation and go to the MSI
remapping component immediately, effectively implementing a "hardware MSI
window". This is what x86 does, the difference being that MSI remapping is
done by the IOMMU on x86, and by the irqchip on ARM.

My current take is that we should keep the current behavior, but I will
try to sort out the different ways of implementing MSIs with virtio-iommu
in the next specification draft.

Thanks,
Jean