Re: Quick start guide on RME emulation

[Mathieu Poirier]

On Mon, 30 Oct 2023 at 04:52, Jean-Philippe Brucker
<jean-philippe@linaro.org> wrote:
>
> On Mon, Oct 23, 2023 at 04:35:35PM -0600, Mathieu Poirier wrote:
> > > > QEMU system emulation
> > > > ---------------------
> > > >
> > > > Repo: https://gitlab.com/qemu-project/qemu.git
> > > >
> > > > Run with:
> > > >
> > > >         qemu-system-aarch64 -M 
> > > > virt,virtualization=on,secure=on,gic-version=3
> > > >                 -M acpi=off -cpu max,x-rme=on,sme=off -m 3G -smp 8
> > > >                 -nographic
> > > >                 -bios tf-a/flash.bin
> > > >                 -kernel linux/arch/arm64/boot/Image
> > > >                 -drive 
> > > > format=raw,if=none,file=buildroot/output/images/rootfs.ext4,id=hd0
> > > >                 -device virtio-blk-pci,drive=hd0
> > > >                 -append root=/dev/vda
> > > >
> > > > To use separate consoles for FW (port 54320), secure payload (54321), 
> > > > host
> > > > (54322) and guest (54323):
> > > >
> > > >                 -nodefaults
> > > >                 -serial tcp:localhost:54320
> > > >                 -serial tcp:localhost:54321
> > > >                 -chardev socket,mux=on,id=hvc0,port=54322,host=localhost
> > > >                 -device virtio-serial-device
> > > >                 -device virtconsole,chardev=hvc0
> > > >                 -chardev socket,mux=on,id=hvc1,port=54323,host=localhost
> > > >                 -device virtio-serial-device
> > > >                 -device virtconsole,chardev=hvc1
> > > >                 -append "root=/dev/vda console=hvc0"
> > > >                 -device virtio-net-pci,netdev=net0 -net user,id=net0
> > >
> > > This argument above gave me an error.  See [1], line 19 to see how I
> > > corrected it.  Notice the "-netdev user,id=net0" rather than "-net
> > > user,id=net0".  Please confirm you have the same on your side.
>
> Yes, I do have '-netdev'
>

Ok

> > >
> > >
> > > Also, "root=/dev/vda" is a duplicate entry.
>
> Right, I left this in case the user wants to keep the default consoles
> (the second '-append' overrides the first one). I'll try to clarify that
> or get rid of it
>
> > >
> > > [1]. https://pastebin.com/KCh8KFD0
> > >
> > > >
> > > >   And connect with 'socat -,rawer TCP-LISTEN:5432x'
> > > >
> > >
> > > There seems to be a syntax error with the above command:
> > >
> > > $ socat -,rawer TCP-LISTEN:54320
> > > 2023/10/19 16:44:55 socat[2799916] E tcsetattr(1, TCSADRAIN,
> > > 0x561add6715c0): Invalid argument
>
> I just encountered this one on Ubuntu, it seems to be a bug in socat that
> has been fixed in newer versions:
> https://bugs.launchpad.net/ubuntu/+source/socat/+bug/1883957
>

Very interesting.  That bug is for 1.7.3.3 and I'm running 1.7.4.1 - I
think that is close enough.

> > >
> > > I ended up writing my own startup script [2] using the same mechanic
> > > as found in OP-TEE.  The "soc_term.spy" file is available here[3].
> > >
> > > Once the QEMU syntax and the console output were fixed I was able to
> > > start the boot stack with the resulting output [4].  The assert, in
> > > function gicv2_driver_init()[5] , is caused by variable gic_version
> > > being set to 0 rather than 1 or 2.  I found that weird because my QEMU
> > > command line [1] explicitly set gic-version=3.  As such I would have
> > > expected to read 3.  And why using gic-version=3 when the code is
> > > specifically looking for a value of 1 or 2?  Any insight would be
> > > appreciated.
> > >
> >
> > I looked into this problem further today and found my way through.
> > Turns out the TF-A defaults to compiling GICv2, and it has to
> > explicitly be told to use GICv3, which I did by adding
> > QEMU_USE_GIC_DRIVER=QEMU_GICV3 when compiling.  As such, the command
> > you have above becomes:
> >
> >         make -j CROSS_COMPILE=aarch64-linux-gnu- PLAT=qemu
> >                  ENABLE_RME=1 DEBUG=1 LOG_LEVEL=40
> > QEMU_USE_GIC_DRIVER=QEMU_GICV3
> >                  RMM=rmm/build-qemu/Debug/rmm.img
> >
> > BL33=edk2/Build/ArmVirtQemuKernel-AARCH64/RELEASE_GCC5/FV/QEMU_EFI.fd
> >                  all fip
> >
> > You problably had to do something like that (or hack the Makefile) but
> > simply forgot about it.
>
> Yes, I do have QEMU_USE_GIC_DRIVER=QEMU_GICV3
>

Ok

> Thank you for helping with this, I'll fix this doc and move it to the wiki
>

I'm almost done folding all that work in the OP-TEE build environment.
I should be in a position to send out build instructions by the end of
my day.  I will include you in the email so that you can try things
out on your side.  We touched on this when I was in Cambridge but
aside from the RMM, the OP-TEE already had all the components of the
CCA boot stack included in a build environment.

>From thereon we can talk about how best to host the code.  My goal is
to have everyone involved working and running the same code base.

While working on this I noticed a few problems, two of which are
probably worth bringing to your attention.

1) When booting the host with more than 3G of RAM, only the boot CPU
is booting properly.  All other CPUs are failing to boot with an angry
looking error message in the kernel boot log.
2) When booting the host with 3G of RAM and giving the Realm above 1G,
the Realm kernel crashes.

Have you seen these issues on your side?  I will likely start working
on #1 next week if you don't already have a solution.

> Thanks,
> Jean
>
> >
> > With that the TF-A is able to complete its
> > initialization and start the Linux kernel in NS world.  Your
> > instructions on how to mount the host's rootFS to gain access to the
> > realm kernel and CPIO work.  From there I was able to start a realm VM
> > and boot it to a command line.  I still haven't seen:
> >
> > [    0.000000] RME: RSI version 13.0 advertised
> >
> > but that is because I haven't enabled early console access in my
> > kernel, something I will do later today.  I also want to start a realm
> > VM using the kvmtools.  From there I want to integrate everything in
> > the OP-TEE build environment and send you a link for you to try.
> >
> > I will get back to you once I have studied the boot stack and the
> > enhancement you did to the various components.  Many thanks for
> > putting this together, it was a huge help.
> >
> > > Regards,
> > > Mathieu
> > >
> > > [2]. https://pastebin.com/XYjUSgSF
> > > [3]. https://github.com/OP-TEE/build/blob/master/soc_term.py
> > > [4]. https://pastebin.com/qnSXrE7z
> > > [5]. 
> > > https://jpbrucker.net/git/tf-a/tree/drivers/arm/gic/v2/gicv2_main.c?h=qemu-rme#n128
> > >
> > > > To share the files needed for booting the guest:
> > > >
> > > >                 -device virtio-9p-device,fsdev=shr0,mount_tag=shr0
> > > >                 -fsdev local,security_model=none,path=.,id=shr0
> > > >
> > > >   And 'mount -t 9p shr0 /mnt' in the host.
> > > >
> > > >
> > > > Then in the host, launch a Realm guest:
> > > >
> > > >         qemu-system-aarch64 -M virt,acpi=off,gic-version=3 -nographic
> > > >                 -cpu host -enable-kvm -smp 2 -m 512M -overcommit 
> > > > mem-lock=on
> > > >                 -kernel /mnt/linux/arch/arm64/boot/Image
> > > >                 -initrd /mnt/buildroot/output/images/rootfs.cpio
> > > >                 -append "console=hvc0 root=/dev/vda"
> > > >                 -M confidential-guest-support=rme0
> > > >                 -object 
> > > > rme-guest,id=rme0,measurement-algo=sha512,num-pmu-counters=6,sve-vector-length=256
> > > >                 -nodefaults -chardev stdio,mux=on,id=hvc0,signal=off
> > > >                 -device virtio-serial-pci -device 
> > > > virtconsole,chardev=hvc0
> > > >                 -device virtio-net-pci,netdev=net0,romfile= -netdev 
> > > > user,id=net0
> > > >                 </dev/hvc1 >/dev/hvc1 &
> > > >         ...
> > > >         [    0.000000] RME: RSI version 13.0 advertised
> > > >