Re: [Qemu-arm] [Qemu-devel] [RFC v2 0/8] VIRTIO-IOMMU device

[Auger Eric]

Hi Bharat,

On 06/07/2017 13:24, Bharat Bhushan wrote:
> 
> 
>> -----Original Message-----
>> From: Jean-Philippe Brucker [mailto:address@hidden
>> Sent: Thursday, July 06, 2017 3:33 PM
>> To: Bharat Bhushan <address@hidden>; Auger Eric
>> <address@hidden>; address@hidden;
>> address@hidden; address@hidden; address@hidden;
>> address@hidden; address@hidden
>> Cc: address@hidden; address@hidden; address@hidden;
>> address@hidden; address@hidden; address@hidden;
>> address@hidden; address@hidden
>> Subject: Re: [Qemu-devel] [RFC v2 0/8] VIRTIO-IOMMU device
>>
>> On 05/07/17 09:49, Bharat Bhushan wrote:>>> Also when setup msi-route
>> kvm_irqchip_add_msi_route() we needed to
>>>> provide the translated address.
>>>>> According to my understanding this is required because kernel does
>>>>> no go
>>>> through viommu translation when generating interrupt, no?
>>>>
>>>> yes this is needed when KVM MSI routes are set up, ie. along with GICV3
>> ITS.
>>>> With GICv2M, qemu direct gsi mapping is used and this is not needed.
>>>>
>>>> So I do not understand your previous sentence saying "MSI interrupts
>>>> works without any change".
>>>
>>> I have almost completed vfio integration with virtio-iommu and now
>>> testing the changes by assigning e1000 device to VM. For this I have
>>> changed virtio-iommu driver to use IOMMU_RESV_MSI rather than sw-msi
>>> and this does not need changed in vfio_get_addr()  and
>>> kvm_irqchip_add_msi_route()
>>
>> I understand you're reserving region 0x08000000-0x08100000 as
>> IOMMU_RESV_MSI instead of IOMMU_RESV_SW_MSI? I think this only
>> works because Qemu places the vgic in that area as well (in hw/arm/virt.c).
>> It's not a coincidence if the addresses are the same, because Eric chose them
>> for the Linux SMMU drivers and I copied them.
>>
>> We can't rely on that behavior, though, it will break MSIs in emulated
>> devices. And if Qemu happens to move the MSI doorbell in future machine
>> revisions, then it would also break VFIO.
> 
> Yes, make sense to me
> 
>>
>> Just for my own understanding -- what happens, I think, is that in Linux
>> iova_reserve_iommu_regions initially reserves the guest-physical doorbell
>> 0x08000000-0x08100000. Then much later, when the device driver requests
>> an MSI, the irqchip driver calls iommu_dma_map_msi_msg with the guest-
>> physical gicv2m address 0x08020000. The function finds the right page in
>> msi_page_list, which was added by cookie_init_hw_msi_region, therefore
>> bypassing the viommu and the GPA gets written in the MSI-X table.
> 
> This means in case tomorrow when qemu changes virt machine address map and 
> vgic-its (its-translator register address) address range does not fall in the 
> msi_page_list then it will allocate a new iova, create mapping in iommu. So 
> this will no longer be identity mapped and fail to work with new qemu?
> 
Yes that's correct.
>>
>> If an emulated device such as virtio-net-pci were to generate an MSI, then
>> Qemu would attempt to access the doorbell written by Linux into the MSI-X
>> table, 0x08020000, and fault because that address wasn't mapped in the
>> viommu.
>>
>> So for VFIO, you either need to translate the MSI-X entry using the viommu,
>> or just assume that the vaddr corresponds to the only MSI doorbell
>> accessible by this device (because how can we be certain that the guest
>> already mapped the doorbell before writing the entry?)
>>
>> For ARM machines it's probably best to stick with IOMMU_RESV_SW_MSI.
>> However, a nice way to use IOMMU_RESV_MSI would be for the virtio-
>> iommu device to advertise identity-mapped/reserved regions, and bypass
>> translation on these regions. Then the driver could reserve those with
>> IOMMU_RESV_MSI.
> 
> Correct me if I did not understood you correctly, today iommu-driver decides 
> msi-reserved region, what if we change this and virtio-iommu device will 
> provide the reserved msi region as per the emulated machine (virt/intel). So 
> virtio-iommu driver will use the address advertised by virtio-iommu device as 
> IOMMU_RESV_MSI. In this case msi-page-list will always have the reserved 
> region for MSI.
> On qemu side, for emulated devices we will let virtio-iommu return same 
> address as translated address as it falls in MSI-reserved page already known 
> to it.

I think what you're proposing here corresponds to the 1st approach that
was followed for PCIe passthrough/MSI on ARM, ie. the userspace was
providing the reserved region base address & size. This was ruled out
and now this region is arbitrarily set by the smmu-driver. At the moment
this means this region cannot contain guest RAM.

> 
> 
>> For x86 we will need such a system, with an added IRQ
>> remapping feature.
> 
> I do not understand x86 MSI interrupt generation, but If above understand is 
> correct, then why we need IRQ remapping for x86?
To me x86 IR corresponds simply corresponds to the ITS MSI controller
modality on ARM. So as you still need vITS along with virtio-iommu on
ARM, you need vIRQ alongs with virtio-iommu on Intel. Does that make sense?

So in any case we need to make sure the guest uses a vITS or vIR to make
sure MSIs are correctly isolated.


> Will the x86 machine emulated in QEMU provides a big address range for MSIs 
> and when actually generating MSI it needed some extra processing 
> (IRQ-remapping processing) before actually generating write transaction for 
> MSI interrupt ? 
My understanding is on x86, the MSI window is fixed and matches
[FEE0_0000h – FEF0_000h]. MSIs are conveyed on a separate address space
than usual DMA accesses. And yes they end up in IR if supported in the HW.

Thanks

Eric
> 
> Thanks
> -Bharat
> 
>>
>> Thanks,
>> Jean