Re: [Qemu-devel] Qemu and 32 PCIe devices

[Paolo Bonzini]

On 09/08/2017 12:56, Laszlo Ersek wrote:
> Allow me one last question -- why (and since when) does each device have
> its own separate address space? Is that related to the virtual IOMMU?

No (though it helps there too).  It's because a device that has
bus-master DMA disabled in the command register cannot see RAM.

So each device has an address space that is just a huge alias for RAM.
The alias is enabled when bus-master DMA is disabled, and disabled when
bus-master DMA is enabled.

> Now that I look at the "info mtree" monitor output of a random VM, I see
> the following "address-space"s:
> - memory
> - I/O
> - cpu-memory
> - bunch of nameless ones, with top level regions called

here they are :)

>   "bus master container"
> - several named "virtio-pci-cfg-as"
> - KVM-SMRAM
> 
> I (sort of) understand MemoryRegions and aliases, but:
> - I don't know why "memory" and "cpu-memory" exist separately, for example,

cpu-memory is also mostly a huge alias to memory.  But some
architectures may see some areas slightly differently when accessed from
devices vs. CPUs.

For example, the 0xFEE00000 area on x86 accesses the APIC when written
from CPUs and triggers MSIs when written from devices.  Not the best
example because we don't use memory vs. cpu-memory to model it, but it's
an example.

Another example was SMM on TCG; when entering SMM, SMRAM used to be
shown/hidden in cpu-memory (and not in memory).  Nowadays TCG uses a
completely separate address space, like KVM-SMRAM in your example above,
but you can see that cpu-memory can come in handy. :)

> - I seem to remember that the "bunch of nameless ones" has not always
> been there? (I could be totally wrong, of course.)

It's always been there IIRC, it comes from 817dcc536898.

Paolo

> ... There is one address_space_init() call in "hw/pci/pci.c", and it
> comes (most recently) from commit 3716d5902d74 ("pci: introduce a bus
> master container", 2017-03-13). The earliest commit that added it seems
> to be 817dcc536898 ("pci: give each device its own address space",
> 2012-10-03). The commit messages do mention IOMMUs.
> 
> Thanks!
> Laszlo
>