Re: [Qemu-devel] [RESEND PATCH 2/6] memory: introduce AddressSpaceOps and IOMMUObject

[David Gibson]

On Wed, Dec 20, 2017 at 02:32:42PM +0800, Liu, Yi L wrote:
> On Mon, Dec 18, 2017 at 10:22:18PM +1100, David Gibson wrote:
> > On Mon, Dec 18, 2017 at 05:17:35PM +0800, Liu, Yi L wrote:
> > > On Mon, Dec 18, 2017 at 05:14:42PM +1100, David Gibson wrote:
> > > > On Thu, Nov 16, 2017 at 04:57:09PM +0800, Liu, Yi L wrote:
[snip]
> > > > So for a typical PAPR setup, the device can access system RAM either
> > > > via DMA in the range 0..1GiB (the "32-bit window") or in the range
> > > > 2^59..2^59+<something> (the "64-bit window").  Typically the 32-bit
> > > > window has mappings dynamically created by drivers, and the 64-bit
> > > > window has all of system RAM mapped 1:1, but that's entirely up to the
> > > > OS, it can map each window however it wants.
> > > > 
> > > > 32-bit devices (or "64 bit" devices which don't actually implement
> > > > enough the address bits) will only be able to use the 32-bit window,
> > > > of course.
> > > > 
> > > > MMIOs of other devices, the "magic" MSI-X addresses belonging to the
> > > > host bridge and other things exist outside those ranges.  Those are
> > > > just the ranges which are used to DMA to RAM.
> > > > 
> > > > Each PE (domain) can see a different version of what's in each
> > > > window.
> > > 
> > > If I'm correct so far. PE actually defines a mapping between an address
> > > range of an address space(aka. iova address space) and an address range
> > > of the system physical address space.
> > 
> > No.  A PE means several things, but basically it is an isolation
> > domain, like an Intel IOMMU domain.  Each PE has an independent set of
> > IOMMU mappings which translate part of the PCI address space to system
> > memory space.
> > 
> > > Then my question is: does each PE define a separate iova address sapce
> > > which is flat from 0 - 2^AW -1, AW is address width? As a reference,
> > > VT-d domain defines a flat address space for each domain.
> > 
> > Partly.  Each PE has an address space which all devices in the PE see.
> > Only some of that address space is mapped to system memory though,
> > other parts are occupied by devices, others are unmapped.
> > 
> > Only the parts mapped by the IOMMU vary between PEs - the other parts
> > of the address space will be identical for all PEs on the host
> 
> Thx, this comment addressed me well. This is different from what we have
> on VT-d.

Really?  That's hard to believe.  I'm pretty sure the VT-d IOMMU must
have a range < 2^64, and anything on the bus outside that range I
expect would be common between all domains.  In particular I'd expect
the BARs for other devices not to be remapped by the IOMMU (though
they may be inaccessible on PCI-E due peer to peer transactions being
blocked).  As well as things above the IOMMU's range, I'd expect the
region for 32-bit BARs to be common between all domains.

[snip]
> > > > > >  IIUC how SVM works, the whole point is that the device
> > > > > > no longer writes into a specific PCI address space.  Instead, it
> > > > > > writes directly into a process address space.  So it seems to me 
> > > > > > more
> > > > > > that SVM should operate at the PCI level, and disassociate the 
> > > > > > device
> > > > > > from the normal PCI address space entirely, rather than hooking up
> > > > > > something via that address space.
> 
> After thinking more, I agree that it is not suitable to hook up something for
> 1st level via the PCI address space. In the time 1st and 2nd level translation
> is exposed to guest, a device would write to multiple address spaces. PCI 
> address
> space is only one of them. I think your reply in another email is a good 
> start,
> let me reply my thoughts under that email.
> 
> Regards,
> Yi L
> 
> > > > > 
> > > > > As Peter replied, we still need the PCI address space, it would be 
> > > > > used
> > > > > to build up the 2nd level page table which would be used in nested
> > > > > translation.
> > > > > 
> > > > > Thanks,
> > > > > Yi L
> > > > > 
> > > > > > 
> > > > > 
> > > > 
> > > 
> > > Regards,
> > > Yi L
> > > 
> > 
> 
> 

-- 
David Gibson                    | I'll have my music baroque, and my code
david AT gibson.dropbear.id.au  | minimalist, thank you.  NOT _the_ _other_
                                | _way_ _around_!
http://www.ozlabs.org/~dgibson

signature.asc
Description: PGP signature