[Qemu-devel] PCI iommu issues

[Benjamin Herrenschmidt]

Hi folks !


I've looked at the intel iommu code to try to figure out how to properly
implement a Power8 "native" iommu model and encountered some issues.

Today "pseries" ppc machine is paravirtualized and so uses a pretty
simplistic iommu model that essentially has one address space per host
bridge.

However, the real HW model I'm working on is closer to Intel in that we
have various tables walked by HW that match an originator RID to what
we called a "PE" (Partitionable Endpoint) to which corresponds an
address space.

So on a given domain, individual functions can have different iommu
address spaces & translation structures, or group of devices etc...
which can all be configured dynamically by the guest OS. This is similar
as far as I understand things to the Intel model though the details of
the implementation are very different.

So I implemented something along the lines of what you guys did for q35
and intel_iommu, and quickly discovered that it doesn't work, which
makes me wonder whether the intel stuff in qemu actually works, or
rather, does it work when adding bridges & switches into the picture.

I basically have two problems but they are somewhat related. Firstly
the way the intel code works is that it creates lazily context
structures that contain the address space, and get associated with
devices when pci_device_iommu_address_space() is called which in
turns calls the bridge iommu_fn which performs the association.

The first problem is that the association is done based on bus/dev/fn
of the device... at a time where bus numbers have not been assigned yet.

In fact, the bus numbers are assigned dynamically by SW, the BIOS
typically, but the OS can renumber things and it's bogus to assume thus
that the RID (bus/dev/fn) of a PCI device/function is fixed. However
that's exactly what the code does as it calls
pci_device_iommu_address_space() once at device instanciation time in
qemu, even before SW had a chance to assign anything.

So as far as I can tell, things will work as long as you are on bus 0
and there is no bridge, otherwise, it's broken by design, unless I'm
missing something...

I've hacked that locally in my code by using the PCIBus * pointer
instead of the bus number to match the device to the iommu context.

The second problem is that pci_device_iommu_address_space(), as it walks
up the hierarchy to find the iommu_fn, drops the original device
information. That means that if a device is below a switch or a p2p
bridge of some sort, once you reach the host bridge top level bus, all
we know is the bus & devfn of the last p2p entity along the path, we
lose the original bus & devfn information.

This is incorrect for that sort of iommu, at least while in the PCIe
domain, as the original RID is carried along with DMA transactions and i
thus needed to properly associate the device/function with a context.

One fix could be to populate the iommu_fn of every bus down the food
chain but that's fairly cumbersome... unless we make the PCI bridges by
default "inherit" from their parent iommu_fn.

Here, I've done a hack locally to keep the original device information
in pci_device_iommu_address_space() but it's not a proper way to do it
either, ultimately, each bridge need to be able to tell whether it
properly forwards the RID information or not, so the bridge itself need
to have some attribute to control that. Typically a PCIe switch or root
complex will always forward the full RID... while most PCI-E -> PCI-X
bridges are busted in that regard. Worse, some bridges forward *some*
bits (partial RID) which is even more broken but I don't know if we can
or even care about simulating it. Thankfully most PCI-X or PCI bridges
will behave properly and make it look like all DMAs are coming from the
bridge itself.

What do you guys reckon is the right approach for both problems ?

Cheers,
Ben.