Re: QEMU unexpectedly closed the monitor

[Alex Bennée]

Itaru Kitayama <itaru.kitayama@linux.dev> writes:

> Hi Alex,
>
>> On Jul 30, 2024, at 22:25, Alex Bennée <alex.bennee@linaro.org> wrote:
>> 
>> Itaru Kitayama <itaru.kitayama@linux.dev> writes:
>> 
>>> Hi,
>>> 
>>> Executing virt-install with the following options:
>>> 
>>> sudo virt-install --machine=virt --arch=aarch64 --name=test8 --disk
>>> path=/var/lib/libvirt/images/jammy.qcow2,format=qcow2,device=disk,bus=virtio,cache=none
>>> --memory=2048 --vcpu=1 --nographic --check all=off --features acpi=off
>>> --import --os-variant=ubuntu22.04 --virt-type kvm --boot
>>> kernel=Image-guest-cca-v4,initrd=rootfs.cpio,kernel_args='earlycon
>>> console=ttyAMA0 rdinit=/sbin/init rw root=/dev/vda acpi=off' --network
>>> none
>>> 
>>> receives an error:
>>> 
>>> Starting install...
>>> ERROR    internal error: QEMU unexpectedly closed the monitor (vm='test8'): 
>>> 2024-07-29T06:23:04.717118Z qemu-system-aarch64: could not load kernel 
>>> '/home/realm/Image-guest-cca-v4’
>>> 
>>> I am not sure how to investigate as the kernel Image (Image-guest-cca-v4) 
>>> can work if it is used directly with the qemu-system-aarch64, like:
>>> 
>>> sudo qemu-system-aarch64 -kernel Image-guest-cca-v4 -initrd
>>> rootfs.cpio -cpu host -M virt,gic-version=3,acpi=off -enable-kvm -smp
>>> 2 -m 2048M -nographic -append 'earlycon console=ttyAMA0
>>> rdinit=/sbin/init rw root=/dev/vda acpi=off' -net none
>> 
>> Is there any way to get virt-install to dump its QEMU command line?
>> There obviously seems to be something off if it works when you build the
>> command line by hand.
>
> Not in a form we wish, but structured way, see the attached log with
> the —debug option.

You can extract the XML into a file and then do:

  virsh domxml-to-native qemu-argv test.xml 

which will give you an excessive command line like this:

  /usr/bin/qemu-system-aarch64 -name guest=test8,debug-threads=on -object 
'{"qom-type":"secret","id":"masterKey0","format":"raw","file":"/home/alex/.config/libvirt/qemu/lib/domain--1-test8/master-key.aes"}'
 -machine 
virt-6.2,usb=off,dump-guest-core=off,gic-version=3,memory-backend=mach-virt.ram 
-accel kvm -cpu host -m 2048 -object 
'{"qom-type":"memory-backend-ram","id":"mach-virt.ram","size":2147483648}' 
-overcommit mem-lock=off -smp 1,sockets=1,cores=1,threads=1 -uuid 
b32a6305-44e2-470e-b514-2c0aff76549b -display none -no-user-config -nodefaults 
-chardev 
socket,id=charmonitor,path=/home/alex/.config/libvirt/qemu/lib/domain--1-test8/monitor.sock,server=on,wait=off
 -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown 
-no-acpi -boot strict=on -kernel /home/realm/Image-v6.10 -initrd 
/home/realm/rootfs.cpio -append 'earlycon console=ttyAMA0 rdinit=/sbin/init rw 
root=/dev/vda acpi=on' -device 
pcie-root-port,port=8,chassis=1,id=pci.1,bus=pcie.0,multifunction=on,addr=0x1 
-device pcie-root-port,port=9,chassis=2,id=pci.2,bus=pcie.0,addr=0x1.0x1 
-device pcie-root-port,port=10,chassis=3,id=pci.3,bus=pcie.0,addr=0x1.0x2 
-device pcie-root-port,port=11,chassis=4,id=pci.4,bus=pcie.0,addr=0x1.0x3 
-device pcie-root-port,port=12,chassis=5,id=pci.5,bus=pcie.0,addr=0x1.0x4 
-device pcie-root-port,port=13,chassis=6,id=pci.6,bus=pcie.0,addr=0x1.0x5 
-device pcie-root-port,port=14,chassis=7,id=pci.7,bus=pcie.0,addr=0x1.0x6 
-device pcie-root-port,port=15,chassis=8,id=pci.8,bus=pcie.0,addr=0x1.0x7 
-device 
pcie-root-port,port=16,chassis=9,id=pci.9,bus=pcie.0,multifunction=on,addr=0x2 
-device pcie-root-port,port=17,chassis=10,id=pci.10,bus=pcie.0,addr=0x2.0x1 
-device pcie-root-port,port=18,chassis=11,id=pci.11,bus=pcie.0,addr=0x2.0x2 
-device pcie-root-port,port=19,chassis=12,id=pci.12,bus=pcie.0,addr=0x2.0x3 
-device pcie-root-port,port=20,chassis=13,id=pci.13,bus=pcie.0,addr=0x2.0x4 
-device pcie-root-port,port=21,chassis=14,id=pci.14,bus=pcie.0,addr=0x2.0x5 
-device qemu-xhci,p2=15,p3=15,id=usb,bus=pci.1,addr=0x0 -device 
virtio-serial-pci,id=virtio-serial0,bus=pci.2,addr=0x0 -blockdev 
'{"driver":"file","filename":"/var/lib/libvirt/images/jammy.qcow2","node-name":"libvirt-1-storage","cache":{"direct":true,"no-flush":false},"auto-read-only":true,"discard":"unmap"}'
 -blockdev 
'{"node-name":"libvirt-1-format","read-only":false,"cache":{"direct":true,"no-flush":false},"driver":"qcow2","file":"libvirt-1-storage"}'
 -device 
virtio-blk-pci,bus=pci.3,addr=0x0,drive=libvirt-1-format,id=virtio-disk0,bootindex=1,write-cache=on
 -chardev pty,id=charserial0 -serial chardev:charserial0 -chardev 
socket,id=charchannel0,path=/home/alex/.config/libvirt/qemu/channel/target/domain--1-test8/org.qemu.guest_agent.0,server=on,wait=off
 -device 
virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.0
 -audiodev '{"id":"audio1","driver":"none"}' -device 
virtio-balloon-pci,id=balloon0,bus=pci.4,addr=0x0 -object 
'{"qom-type":"rng-random","id":"objrng0","filename":"/dev/urandom"}' -device 
virtio-rng-pci,rng=objrng0,id=rng0,bus=pci.5,addr=0x0 -sandbox 
on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny -msg 
timestamp=on

You then need to manually strip out all the various chardevs for libvirt
control sockets and you can an equivalent command line you can run from
the console. One thing that did jump out as a bit weird to me was:

 -rtc base=utc -no-shutdown -no-acpi -boot strict=on \
 -kernel /home/realm/Image-v6.10 \
 -initrd /home/realm/rootfs.cpio \
 -append 'earlycon console=ttyAMA0 rdinit=/sbin/init rw root=/dev/vda acpi=on'

which seems to be both disabling ACPI and then enabling it in the kernel
options. Maybe that is a firmware thing?

>
> Itaru.
>
> [2. application/octet-stream; log]...
>
>
>> 
>>> 
>>> Userland is Ubuntu 24.04, and the host and guest kernels are for the 
>>> protected VM (Realm) execution.
>>> Any help would be appreciated.
>>> 
>>> Thanks,
>>> Itaru.
>> 
>> -- 
>> Alex Bennée
>> Virtualisation Tech Lead @ Linaro

-- 
Alex Bennée
Virtualisation Tech Lead @ Linaro