[Qemu-devel] Supporting multiple CPU AddressSpaces and memory transaction attributes

[Peter Maydell]

One of the parts of the ARM TrustZone/Security Extensions
which the patchsets we've seen so far haven't attempted to
tackle is the problem of Secure vs NonSecure memory accesses.
Architecturally, every memory transaction should have
an S/NS bit accompanying the physical address, effectively
making an extra address bit (you could in theory have
completely different physical memory maps for S and NS,
though usually they're similar). We can't fake this up
by having the device call back into the CPU to check its
current status, because the CPU can make NS accesses
when it is in the Secure world (controlled by page table bits).

We have some other cases where devices would really
like to have some kind of "memory transaction attributes"
information:
 * watchpoints on ARM need to know whether the access
   was userspace or system (ie which mmu_idx it was).
   [The LDRT/STRT instructions which let the kernel
   do userspace-privilege accesses mean that the current
   state of the CPU isn't sufficient to determine this.]
   They'll also want to know the S/NS info.
 * The GIC wants to provide a different set of registers
   to each CPU (currently we do this with a hacky use of
   current_cpu), as do some other devices.

Paolo also mentioned that x86 SMM has some situations
where devices need to be only visible in some cases.

(Another oddball usecase is the Cortex-M split I and D
bus for low memory, where instruction and data accesses
go out via different buses and might map to different things,
but for now I think I'm happy to ignore this as more a
theoretical question than a practical one...)

Here's one idea which deals with some of these...

We introduce the concept of memory transaction attributes,
which are a guest-CPU specific bunch of bits (say, a
uint32_t). We also allow the CPU to have more than one
AddressSpace, and have the guest CPU code provide
a function returning the AddressSpace to use for a
given set of transaction attributes. For ARM we'd
put all of (S/NS, mmu_idx, CPU number) into the attributes,
use an AddressSpace each for S and NS, and use the
S/NS bit in the attributes to select the AddressSpace
The default is that we have one AddressSpace and always
use that, ie the transaction attributes are ignored.
(Maybe the function should return an integer index
into a cpu->as[] array?)

tlb_set_page() takes an extra argument specifying the
transaction attributes. For RAM accesses we can just
immediately use this to get the AddressSpace to pass
to address_space_translate_for_iotlb(). For IO accesses
we need to stash the attributes in the iotlb[], which
means extending that from an array of hwaddrs to an
array of struct {hwaddr, attributes}, which is easy enough.
Then the io_read/write glue functions in softmmu_template.h
can fish the attributes out of the iotlb and use them to
pick the AddressSpace to pass to iotlb_to_region().
More importantly, we can arrange to pass them through
to the device read/write callbacks (either directly,
or indirectly by saving them in the CPU struct like we
do for mem_io_vaddr; since changing the prototypes on
every device read and write callback would be insane
we probably want to have fields in MemoryRegionOps for
read_with_attrs and write_with_attrs function pointers).

We would need APIs so bus masters other than CPUs can
specify the transaction attributes (and it's probably
a good idea for the guest CPU to arrange its attribute
bit definitions so that "0" means "no attribute info",
to account for legacy bus masters).

The watchpoint read/write callbacks would just stuff
the transaction attribute word for the access that
triggered the watchpoint into the CPUWatchpoint struct
so that target-specific code can look at it later.

This combination of things would let us handle both
"some devices and some RAM are only visible in
the Secure address space" (by having the board
construct the S AddressSpace with them and the NS
without them) and also "some devices are TrustZone
aware and behave differently" (by having the device
read/write callbacks look at the attributes word).

Thoughts?

-- PMM