Re: [Qemu-devel] [PATCH] pcie-root-port: let it has higher migrate priority

[Marcel Apfelbaum]

On 02/02/2018 11:56, Peter Xu wrote:
> On Thu, Feb 01, 2018 at 07:51:31PM +0000, Dr. David Alan Gilbert wrote:
>> * Peter Xu (address@hidden) wrote:
>>> In the past, we prioritized IOMMU migration so that we have such a
>>> priority order:
>>>
>>>     IOMMU > PCI Devices
>>>
>>> When migrating a guest with both vIOMMU and pcie-root-port, we'll always
>>> migrate vIOMMU first, since pcie-root-port will be seen to have the same
>>> priority of general PCI devices.
>>>
>>> That's problematic.
>>>
>>> The thing is that PCI bus number information is stored in the root port,
>>> and that is needed by vIOMMU during post_load(), e.g., to figure out
>>> context entry for a device.  If we don't have correct bus numbers for
>>> devices, we won't be able to recover device state of the DMAR memory
>>> regions, and things will be messed up.
>>>
>>> So let's boost the PCIe root ports to be even with higher priority:
>>>
>>>    PCIe Root Port > IOMMU > PCI Devices
>>>
>>> A smoke test shows that this patch fixes bug 1538953.
>>
>> Two questions (partially overlapping with what I replied to Michaels):
>>   a) What happens with multiple IOMMUs?
> 
> If there are more IOMMUs, then the patch will let all the vIOMMUs be
> migrated after pcie root ports.
> 
> But a more true answer is that: I don't really know. :)
> 
> Because I even don't know how multiple vIOMMUs will coop with each
> other, especially nested. 

I am not aware of "nested" IOMMUs. Multiple IOMMUs work together
by dividing the bus ranges, when each of them declares in the
corresponding ACPI table the bus/device/range is in charge of.

However there was a kernel bug some time ago preventing several
IOMMUs to work together, I am not sure the problem is solved yet.


 In nested case, maybe there will be
> dependency between vIOMMUs, but I'll avoid thinking about that until
> we support more than one vIOMMUs.
> 
>>   b) What happens with multiple root ports?
> 
> Same answer as previous one: all of them will be migrated before any
> vIOMMUs.
> 
> Note that IMHO we don't care which pcie root port is migrated first -
> IMHO they should not depend on each other, but Marcel may correct me.
> 

Right, each Root Port is independent from each other.

Thanks,
Marcel

>>   c) How correct is this ordering on different implementations 
>>     (e.g. ARM/Power/etc)
> 
> Currently it won't affect since Intel IOMMU is the only user for
> MIG_PRI_IOMMU.  After SMMU is merged it may affect (if it uses this
> bit), but IMHO it's fine too as long as pcie root ports won't depend
> on anything related to SMMU.
> 
> Thanks,
>