Re: [Qemu-devel] Supporting multiple CPU AddressSpaces and memory transaction attributes

[Peter Crosthwaite]

On Fri, Sep 5, 2014 at 3:47 AM, Peter Maydell <address@hidden> wrote:
> One of the parts of the ARM TrustZone/Security Extensions
> which the patchsets we've seen so far haven't attempted to
> tackle is the problem of Secure vs NonSecure memory accesses.
> Architecturally, every memory transaction should have
> an S/NS bit accompanying the physical address, effectively
> making an extra address bit (you could in theory have
> completely different physical memory maps for S and NS,
> though usually they're similar). We can't fake this up
> by having the device call back into the CPU to check its
> current status, because the CPU can make NS accesses
> when it is in the Secure world (controlled by page table bits).
>
> We have some other cases where devices would really
> like to have some kind of "memory transaction attributes"
> information:
>  * watchpoints on ARM need to know whether the access
>    was userspace or system (ie which mmu_idx it was).
>    [The LDRT/STRT instructions which let the kernel
>    do userspace-privilege accesses mean that the current
>    state of the CPU isn't sufficient to determine this.]
>    They'll also want to know the S/NS info.
>  * The GIC wants to provide a different set of registers
>    to each CPU (currently we do this with a hacky use of
>    current_cpu), as do some other devices.
>
> Paolo also mentioned that x86 SMM has some situations
> where devices need to be only visible in some cases.
>
> (Another oddball usecase is the Cortex-M split I and D
> bus for low memory, where instruction and data accesses
> go out via different buses and might map to different things,
> but for now I think I'm happy to ignore this as more a
> theoretical question than a practical one...)
>

Similar problem that I ran into was trying to differentiate I from D
for unassigned memory accesses. I'm trying to correctly implement
prefetch and data aborts for unassigned but the memory API has no
information on whether an access is I or D. Attributes could solve
this.

> Here's one idea which deals with some of these...
>
> We introduce the concept of memory transaction attributes,
> which are a guest-CPU specific bunch of bits (say, a
> uint32_t). We also allow the CPU to have more than one
> AddressSpace,

Or any master for that matter. I have an example of this for a dma
device already:

http://lists.gnu.org/archive/html/qemu-devel/2014-06/msg00370.html

 and have the guest CPU code provide
> a function returning the AddressSpace to use for a
> given set of transaction attributes. For ARM we'd
> put all of (S/NS, mmu_idx, CPU number) into the attributes,

Many buses have master ID or transaction ID for master identification,
so I think the concept of CPU number should be softened to just
master-id.

> use an AddressSpace each for S and NS, and use the
> S/NS bit in the attributes to select the AddressSpace
> The default is that we have one AddressSpace and always
> use that, ie the transaction attributes are ignored.
> (Maybe the function should return an integer index
> into a cpu->as[] array?)
>
> tlb_set_page() takes an extra argument specifying the
> transaction attributes. For RAM accesses we can just
> immediately use this to get the AddressSpace to pass
> to address_space_translate_for_iotlb(). For IO accesses
> we need to stash the attributes in the iotlb[], which
> means extending that from an array of hwaddrs to an
> array of struct {hwaddr, attributes}, which is easy enough.
> Then the io_read/write glue functions in softmmu_template.h
> can fish the attributes out of the iotlb and use them to
> pick the AddressSpace to pass to iotlb_to_region().
> More importantly, we can arrange to pass them through
> to the device read/write callbacks (either directly,
> or indirectly by saving them in the CPU struct like we
> do for mem_io_vaddr; since changing the prototypes on
> every device read and write callback would be insane
> we probably want to have fields in MemoryRegionOps for
> read_with_attrs and write_with_attrs function pointers).
>
> We would need APIs so bus masters other than CPUs can
> specify the transaction attributes (and it's probably
> a good idea for the guest CPU to arrange its attribute
> bit definitions so that "0" means "no attribute info",
> to account for legacy bus masters).
>
> The watchpoint read/write callbacks would just stuff
> the transaction attribute word for the access that
> triggered the watchpoint into the CPUWatchpoint struct
> so that target-specific code can look at it later.
>
> This combination of things would let us handle both
> "some devices and some RAM are only visible in
> the Secure address space" (by having the board
> construct the S AddressSpace with them and the NS
> without them) and also "some devices are TrustZone
> aware and behave differently" (by having the device
> read/write callbacks look at the attributes word).
>
> Thoughts?
>

The transaction attributes concept you are proposing sounds orthogonal
to the multi-address space work I have been working on, so I guess I
should proceed with that series to solve some of the multi-address
space requirements independently of the transaction attrs extension.

Overall sounds ok, more than just addr and data needs to come on IO
bus transactions and there's lots of features that require it.

Regards,
Peter

> -- PMM
>