Re: [PATCH v3 00/26] arm: Run Arm CCA VMs with KVM

[Gavin Shan]

Hi Jean,

On 11/26/24 5:55 AM, Jean-Philippe Brucker wrote:
> This series enables running confidential VMs on Arm CCA. The host KVM
> support is progressing but still under discussion [1], so there is no
> urgency to upstream this series. I'm sending this new version to give a
> status update, and also to discuss remote attestation below.
>
> Since v2 [2] I addressed comments on the QAPI patches. The support for
> running Linux in a Realm will be in Linux v6.13 [3], so the guest-facing
> interface is now stable. One important change since v2 is the requirement
> to initialize the whole GPA space in RMM before boot, which we do in patch
> 9. The 'earlycon' kernel parameter now requires an unprotected address
> parameter (see Documentation/arch/arm64/arm-cca.rst in Linux v6.13).
>
> Documentation to try this series out:
> https://linaro.atlassian.net/wiki/spaces/QEMU/pages/29051027459/Building+an+RME+stack+for+QEMU

I followed the instructions, but encountering some issues. First of all, the
recommended TF-RMM (branch: cca/v4) fails to be built. After switching to the
latest upstream TF-RMM, it fails to be initialized.

Note: the combination of the upstream TF-A + upstream TF-RMM + upstream EDK2
worked fine for the 'fvp' emulator.

TF-RMM fails to be built
========================
$ git clone https://git.codelinaro.org/linaro/dcap/rmm.git tf-rmm
$ cd tf-rmm
$ git checkout origin/cca/v4 -b cca/v4
$ git submodule update --init --recursive
$ cmake -DCMAKE_BUILD_TYPE=Debug -DRMM_CONFIG=qemu_virt_defcfg -B build-qemu
-- The C compiler identification is GNU 11.5.0
-- The CXX compiler identification is GNU 11.5.0
-- The ASM compiler identification is GNU
-- Found assembler: /usr/bin/gcc
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - failed
-- Check for working C compiler: /usr/bin/gcc
-- Check for working C compiler: /usr/bin/gcc - broken
CMake Error at /usr/share/cmake/Modules/CMakeTestCCompiler.cmake:67 (message):
  The C compiler

    "/usr/bin/gcc"

  is not able to compile a simple test program.

  It fails with the following output:

    Change Dir: 
/home/gshan/sandbox/qemu/host/tf-rmm/build-qemu/CMakeFiles/CMakeScratch/TryCompile-Aab6zP
    
    Run Build Command(s):/usr/bin/cmake -E env VERBOSE=1 /usr/bin/gmake -f Makefile cmTC_87ded/fast && /usr/bin/gmake  -f CMakeFiles/cmTC_87ded.dir/build.make CMakeFiles/cmTC_87ded.dir/build
    gmake[1]: Entering directory 
'/home/gshan/sandbox/qemu/host/tf-rmm/build-qemu/CMakeFiles/CMakeScratch/TryCompile-Aab6zP'
    Building C object CMakeFiles/cmTC_87ded.dir/testCCompiler.c.obj
    /usr/bin/gcc   -fno-common -ffunction-sections -fdata-sections -Wall 
-Werror -gdwarf-4 -ffreestanding -mbranch-protection=standard 
-mgeneral-regs-only -mstrict-align -fpie  -o 
CMakeFiles/cmTC_87ded.dir/testCCompiler.c.obj -c 
/home/gshan/sandbox/qemu/host/tf-rmm/build-qemu/CMakeFiles/CMakeScratch/TryCompile-Aab6zP/testCCompiler.c
    gcc: error: unrecognized command-line option ‘-mbranch-protection=standard’
    gcc: error: unrecognized command-line option ‘-mstrict-align’; did you mean 
‘-Wstrict-aliasing’?
    gmake[1]: *** [CMakeFiles/cmTC_87ded.dir/build.make:78: 
CMakeFiles/cmTC_87ded.dir/testCCompiler.c.obj] Error 1
    gmake[1]: Leaving directory 
'/home/gshan/sandbox/qemu/host/tf-rmm/build-qemu/CMakeFiles/CMakeScratch/TryCompile-Aab6zP'
    gmake: *** [Makefile:127: cmTC_87ded/fast] Error 2

  CMake will not be able to correctly generate this project.
Call Stack (most recent call first):
  CMakeLists.txt:51 (project)


-- Configuring incomplete, errors occurred!

Upstream TF-RMM fails to be initialized
========================================
$ git clone git@github.com:TF-RMM/tf-rmm.git tf-rmm
$ cd tf-rmm
$ git submodule update --init --recursive
$ export CROSS_COMPILE=aarch64-none-elf-
$ cmake -DCMAKE_BUILD_TYPE=Debug -DRMM_CONFIG=qemu_virt_defcfg -B build-qemu
$ cmake --build build-qemu

$ cd ..
$ git clone git@github.com:tianocore/edk2.git edk2
$ cd edk2
$ git submodule update --init --recursive
$ source edksetup.sh
$ make -j -C BaseTools
$ export GCC5_AARCH64_PREFIX=aarch64-linux-gnu-
$ build -b RELEASE -a AARCH64 -t GCC5 -p ArmVirtPkg/ArmVirtQemuKernel.dsc

$ cd ..
$ git@github.com:ARM-software/arm-trusted-firmware.git tf-a
$ make -j CROSS_COMPILE=aarch64-linux-gnu-     \
  PLAT=qemu ENABLE_RME=1 DEBUG=1 LOG_LEVEL=40  \
  QEMU_USE_GIC_DRIVER=QEMU_GICV3               \
  RMM=../rmm/build-qemu/Debug/rmm.img          \
  BL33=../edk2/Build/ArmVirtQemuKernel-AARCH64/RELEASE_GCC5/FV/QEMU_EFI.fd all 
fip
$ dd if=build/qemu/debug/bl1.bin of=flash.bin
$ dd if=build/qemu/debug/fip.bin of=flash.bin seek=64 bs=4096

$ ${HOST_PATH}/qemu/build/qemu-system-aarch64                         \
-M virt,virtualization=on,secure=on,gic-version=3,acpi=off            \
-cpu max,x-rme=on -m 64G -smp 8                                       \
-serial mon:stdio -monitor none -nographic -nodefaults                \
-bios ${HOST_PATH}/tf-a/flash.bin                                     \
-kernel ${HOST_PATH}/linux/arch/arm64/boot/Image                      \
-initrd ${HOST_PATH}/buildroot/output/images/rootfs.cpio.xz           \
-device pcie-root-port,bus=pcie.0,chassis=1,id=pcie.1                 \
-device pcie-root-port,bus=pcie.0,chassis=2,id=pcie.2                 \
-device pcie-root-port,bus=pcie.0,chassis=3,id=pcie.3                 \
-device pcie-root-port,bus=pcie.0,chassis=4,id=pcie.4                 \
-device virtio-9p-device,fsdev=shr0,mount_tag=shr0                    \
-fsdev local,security_model=none,path=${GUEST_PATH},id=shr0           \
-netdev tap,id=tap1,script=/etc/qemu-ifup,downscript=/etc/qemu-ifdown \
-device virtio-net-pci,bus=pcie.2,netdev=tap1,mac=78:ac:44:2b:43:f0
  :
  :
NOTICE:  Booting Trusted Firmware
NOTICE:  BL1: v2.12.0(debug):ad6310a66
NOTICE:  BL1: Built : 22:09:22, Dec  7 2024
INFO:    BL1: RAM 0xe0ee000 - 0xe0f6000
INFO:    BL1: Loading BL2
INFO:    Loading image id=1 at address 0xe05b000
INFO:    Image id=1 loaded: 0xe05b000 - 0xe0652b9
NOTICE:  BL1: Booting BL2
INFO:    Entry point address = 0xe05b000
INFO:    SPSR = 0x3cd
INFO:    GPT: Boot Configuration
INFO:      PPS/T:     0x2/40
INFO:      PGS/P:     0x0/12
INFO:      L0GPTSZ/S: 0x0/30
INFO:      PAS count: 6
INFO:      L0 base:   0xeefc000
INFO:    Enabling Granule Protection Checks
NOTICE:  BL2: v2.12.0(debug):ad6310a66
NOTICE:  BL2: Built : 22:09:23, Dec  7 2024
INFO:    BL2: Doing platform setup
INFO:    Reserved RMM memory [0x40100000, 0x418fffff] in Device tree
INFO:    BL2: Loading image id 3
INFO:    Loading image id=3 at address 0xe090000
INFO:    Image id=3 loaded: 0xe090000 - 0xe0a20c4
INFO:    BL2: Loading image id 35
INFO:    Loading image id=35 at address 0x40100000
INFO:    Image id=35 loaded: 0x40100000 - 0x40303a00
INFO:    BL2: Loading image id 5
INFO:    Loading image id=5 at address 0x60000000
INFO:    Image id=5 loaded: 0x60000000 - 0x60200000
NOTICE:  BL2: Booting BL31
INFO:    Entry point address = 0xe090000
INFO:    SPSR = 0x3cd
NOTICE:  BL31: v2.12.0(debug):ad6310a66
NOTICE:  BL31: Built : 22:09:23, Dec  7 2024
INFO:    GICv3 without legacy support detected.
INFO:    ARM GICv3 driver initialized in EL3
INFO:    Maximum SPI INTID supported: 287
INFO:    BL31: Initializing runtime services
INFO:    RMM setup done.
INFO:    BL31: Initializing RMM
INFO:    RMM init start.
ERROR:   RMM init failed: -7               <<<<< RMM initialization failed
WARNING: BL31: RMM initialization failed
INFO:    BL31: Preparing for EL3 exit to normal world
INFO:    Entry point address = 0x60000000
INFO:    SPSR = 0x3c9
UEFI firmware (version  built at 22:03:35 on Dec  7 2024)
 :
EFI stub: Booting Linux Kernel...
EFI stub: Loaded initrd from LINUX_EFI_INITRD_MEDIA_GUID device path
EFI stub: Using DTB from configuration table
EFI stub: Exiting boot services...
ERROR:   RMM Failed to initialize. Ignoring for CPU1
ERROR:   RMM Failed to initialize. Ignoring for CPU2
ERROR:   RMM Failed to initialize. Ignoring for CPU3
ERROR:   RMM Failed to initialize. Ignoring for CPU4
ERROR:   RMM Failed to initialize. Ignoring for CPU5
ERROR:   RMM Failed to initialize. Ignoring for CPU6
ERROR:   RMM Failed to initialize. Ignoring for CPU7
[    0.000000] Booting Linux on physical CPU 0x0000000000 [0x000f0510]

Thanks,
Gavin