Re: [Qemu-arm] [Qemu-devel] [RFC v2 0/8] VIRTIO-IOMMU device

[Bharat Bhushan]

> -----Original Message-----
> From: Jean-Philippe Brucker [mailto:address@hidden
> Sent: Thursday, July 06, 2017 3:33 PM
> To: Bharat Bhushan <address@hidden>; Auger Eric
> <address@hidden>; address@hidden;
> address@hidden; address@hidden; address@hidden;
> address@hidden; address@hidden
> Cc: address@hidden; address@hidden; address@hidden;
> address@hidden; address@hidden; address@hidden;
> address@hidden; address@hidden
> Subject: Re: [Qemu-devel] [RFC v2 0/8] VIRTIO-IOMMU device
> 
> On 05/07/17 09:49, Bharat Bhushan wrote:>>> Also when setup msi-route
> kvm_irqchip_add_msi_route() we needed to
> >> provide the translated address.
> >>> According to my understanding this is required because kernel does
> >>> no go
> >> through viommu translation when generating interrupt, no?
> >>
> >> yes this is needed when KVM MSI routes are set up, ie. along with GICV3
> ITS.
> >> With GICv2M, qemu direct gsi mapping is used and this is not needed.
> >>
> >> So I do not understand your previous sentence saying "MSI interrupts
> >> works without any change".
> >
> > I have almost completed vfio integration with virtio-iommu and now
> > testing the changes by assigning e1000 device to VM. For this I have
> > changed virtio-iommu driver to use IOMMU_RESV_MSI rather than sw-msi
> > and this does not need changed in vfio_get_addr()  and
> > kvm_irqchip_add_msi_route()
> 
> I understand you're reserving region 0x08000000-0x08100000 as
> IOMMU_RESV_MSI instead of IOMMU_RESV_SW_MSI? I think this only
> works because Qemu places the vgic in that area as well (in hw/arm/virt.c).
> It's not a coincidence if the addresses are the same, because Eric chose them
> for the Linux SMMU drivers and I copied them.
> 
> We can't rely on that behavior, though, it will break MSIs in emulated
> devices. And if Qemu happens to move the MSI doorbell in future machine
> revisions, then it would also break VFIO.

Yes, make sense to me

> 
> Just for my own understanding -- what happens, I think, is that in Linux
> iova_reserve_iommu_regions initially reserves the guest-physical doorbell
> 0x08000000-0x08100000. Then much later, when the device driver requests
> an MSI, the irqchip driver calls iommu_dma_map_msi_msg with the guest-
> physical gicv2m address 0x08020000. The function finds the right page in
> msi_page_list, which was added by cookie_init_hw_msi_region, therefore
> bypassing the viommu and the GPA gets written in the MSI-X table.

This means in case tomorrow when qemu changes virt machine address map and 
vgic-its (its-translator register address) address range does not fall in the 
msi_page_list then it will allocate a new iova, create mapping in iommu. So 
this will no longer be identity mapped and fail to work with new qemu?

> 
> If an emulated device such as virtio-net-pci were to generate an MSI, then
> Qemu would attempt to access the doorbell written by Linux into the MSI-X
> table, 0x08020000, and fault because that address wasn't mapped in the
> viommu.
> 
> So for VFIO, you either need to translate the MSI-X entry using the viommu,
> or just assume that the vaddr corresponds to the only MSI doorbell
> accessible by this device (because how can we be certain that the guest
> already mapped the doorbell before writing the entry?)
> 
> For ARM machines it's probably best to stick with IOMMU_RESV_SW_MSI.
> However, a nice way to use IOMMU_RESV_MSI would be for the virtio-
> iommu device to advertise identity-mapped/reserved regions, and bypass
> translation on these regions. Then the driver could reserve those with
> IOMMU_RESV_MSI.

Correct me if I did not understood you correctly, today iommu-driver decides 
msi-reserved region, what if we change this and virtio-iommu device will 
provide the reserved msi region as per the emulated machine (virt/intel). So 
virtio-iommu driver will use the address advertised by virtio-iommu device as 
IOMMU_RESV_MSI. In this case msi-page-list will always have the reserved region 
for MSI.
On qemu side, for emulated devices we will let virtio-iommu return same address 
as translated address as it falls in MSI-reserved page already known to it.


> For x86 we will need such a system, with an added IRQ
> remapping feature.

I do not understand x86 MSI interrupt generation, but If above understand is 
correct, then why we need IRQ remapping for x86?
Will the x86 machine emulated in QEMU provides a big address range for MSIs and 
when actually generating MSI it needed some extra processing (IRQ-remapping 
processing) before actually generating write transaction for MSI interrupt ? 

Thanks
-Bharat

> 
> Thanks,
> Jean