Re: [PATCH v9 2/3] Acceptance test: add "boot_linux" tests

[Andrew Jones]

On Thu, Feb 20, 2020 at 02:52:45PM -0500, Cleber Rosa wrote:
> On Thu, Feb 20, 2020 at 01:49:40PM -0300, Wainer dos Santos Moschetta wrote:
> > On 2/19/20 11:06 PM, Cleber Rosa wrote:
> > > +
> > > +    def test_virt_tcg(self):
> > > +        """
> > > +        :avocado: tags=accel:tcg
> > > +        :avocado: tags=cpu:cortex-a53
> > > +        """
> > > +        if not tcg_available(self.qemu_bin):
> > > +            self.cancel(TCG_NOT_AVAILABLE)
> > > +        self.vm.add_args("-accel", "tcg")
> > > +        self.vm.add_args('-cpu', 'cortex-a53')

Should use '-cpu max'. gic-version will be '2' by default, which is good
for tcg, but I would probably add an explicit '-machine gic-version=2'
anyway.

> > > +        self.add_common_args()
> > > +        self.launch_and_wait()
> > > +
> > > +    def test_virt_kvm(self):
> > > +        """
> > > +        :avocado: tags=accel:kvm
> > > +        :avocado: tags=cpu:host
> > > +        """
> > > +        if not kvm_available(self.arch, self.qemu_bin):
> > > +            self.cancel(KVM_NOT_AVAILABLE)
> > > +        self.vm.add_args("-accel", "kvm")
> > > +        self.vm.add_args("-cpu", "host")

kvm needs '-machine gic-version=max' and could use '-cpu max' too,
because, for kvm, CPU::max == CPU::host

> > > +        self.add_common_args()
> > > +        self.launch_and_wait()
> > 
> > 
> > For aarch64 tests it seems '-cpu max' is the best choice. See in
> > https://www.mail-archive.com/address@hidden/msg672755.html
> > 
> >
> 
> +drew
> 
> Thanks for pointing that out.  There's one thing, though, which I can
> not agree on.  And I know that Drew is an expert on the matter, which
> makes it harder to disagree on... but, I've got results which clearly
> indicate that *not using* the gic-version machine parameter still gets
> me KVM:

The gic-version machine property is completely independent of whether kvm
or tcg is selected. However, if you select kvm and a gic-version that is
incompatible with the host then the guest will not start. If gic-version
is not specified it defaults to '2'. Below the kvm guest started
successfully because it happened to be started on a host with gicv2 or
on a host with gicv3 that supports gicv2-on-v3 (which is an optional
feature that doesn't appear to be getting implemented in modern AArch64
servers). The test below would have failed to start the guest on a host
with only gicv3.

When kvm is in use, one must use gic-version=max in order to automatically
select the latest host-compatible gic version, or the guest will not start
on all hosts.

> 
> ./tests/venv/bin/avocado run 
> tests/acceptance/boot_linux.py:BootLinuxAarch64.test_virt_kvm
> JOB ID     : 21a394b884b474ceee0a045b3e74f98da0aee023
> JOB LOG    : 
> /home/cleber/avocado/job-results/job-2020-02-20T14.28-21a394b/job.log
>  (1/1) tests/acceptance/boot_linux.py:BootLinuxAarch64.test_virt_kvm: PASS 
> (35.10 s)
> RESULTS    : PASS 1 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | 
> CANCEL 0
> JOB TIME   : 35.87 s
> 
> VM launch command:
>    aarch64-softmmu/qemu-system-aarch64
>    -display none
>    -vga none
>    -chardev socket,id=mon,path=/var/tmp/tmpntz_r_h7/qemu-18331-monitor.sock
>    -mon chardev=mon,mode=control
>    -machine virt
>    -chardev 
> socket,id=console,path=/var/tmp/tmpntz_r_h7/qemu-18331-console.sock,server,nowait
>    -serial chardev:console
>    -smp 2
>    -m 1024
>    -drive 
> file=/var/tmp/avocado_u9jm04di/avocado_job_28oth9kk/1-tests_acceptance_boot_linux.py_BootLinuxAarch64.test_virt_kvm/Fedora-Cloud-Base-31-1.9.aarch64-05265df5.qcow2
>  -drive 
> file=/var/tmp/avocado_u9jm04di/avocado_job_28oth9kk/1-tests_acceptance_boot_linux.py_BootLinuxAarch64.test_virt_kvm/cloudinit.iso,format=raw
>    -accel kvm
>    -cpu host
>    -bios /home/cleber/build/qemu/pc-bios/edk2-aarch64-code.fd
>    -device virtio-rng-pci,rng=rng0
>    -object rng-random,id=rng0,filename=/dev/urandom
> 
> Guest boot messages shows:
> [    1.538955] systemd[1]: Detected virtualization kvm.
> [    1.539828] systemd[1]: Detected architecture arm64.
> 
> This is in contrast with:
> 
> ./tests/venv/bin/avocado run 
> tests/acceptance/boot_linux.py:BootLinuxAarch64.test_virt_tcg 
> JOB ID     : 90b9412f700e52428b59e97719496c30b4f54435
> JOB LOG    : 
> /home/cleber/avocado/job-results/job-2020-02-20T14.32-90b9412/job.log
>  (1/1) tests/acceptance/boot_linux.py:BootLinuxAarch64.test_virt_tcg: PASS 
> (581.14 s)
> RESULTS    : PASS 1 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | 
> CANCEL 0
> JOB TIME   : 581.93 s
> 
> VM launch command:
>    aarch64-softmmu/qemu-system-aarch64
>    -display none
>    -vga none
>    -chardev socket,id=mon,path=/var/tmp/tmpa6i4livg/qemu-18498-monitor.sock
>    -mon chardev=mon,mode=control
>    -machine virt
>    -chardev 
> socket,id=console,path=/var/tmp/tmpa6i4livg/qemu-18498-console.sock,server,nowait
>    -serial chardev:console
>    -smp 2
>    -m 1024
>    -drive 
> file=/var/tmp/avocado_slcj2x9e/avocado_job_x5u__309/1-tests_acceptance_boot_linux.py_BootLinuxAarch64.test_virt_tcg/Fedora-Cloud-Base-31-1.9.aarch64-5b006a2f.qcow2
>  -drive 
> file=/var/tmp/avocado_slcj2x9e/avocado_job_x5u__309/1-tests_acceptance_boot_linux.py_BootLinuxAarch64.test_virt_tcg/cloudinit.iso,format=raw
>    -accel tcg
>    -cpu cortex-a53
>    -bios /home/cleber/build/qemu/pc-bios/edk2-aarch64-code.fd
>    -device virtio-rng-pci,rng=rng0
>    -object rng-random,id=rng0,filename=/dev/urandom'
> 
> Guest boot messages shows:
> [   28.606310] systemd[1]: Detected virtualization qemu.
> [   28.607861] systemd[1]: Detected architecture arm64.
> 
> And with regards to the CPU type, IIRC, "max" will fallback to the
> best CPU on TCG mode.

No, it doesn't fallback. It implements the cortex-a57 and enables all
optional CPU features. Why was the cortex-a53 chosen for the tests?

> As a general best practice in testing, I'd
> rather not have this dynamic aspect where we can avoid it.  Looks like
> with TCG we can set it to one CPU and validate that the guests work on
> that configuration.

The only dynamic aspect is that as new CPU features are implemented
they'll get enabled. Personally I'd prefer the tests run with the latest
code enabled in order to find the latest bugs.

> 
> IIUC, by using either "-cpu host" or "-cpu max" for KVM, we may end up
> having the same test PASS or FAIL because of the (dynamic) host CPU.
> That's not ideal for testing purposes, but given it's outside of our
> control, do best we can do is keep track of the host CPU (via Avocado's
> sysinfo collection).

Right. AArch64 KVM must use '-cpu host' (or equivalently '-cpu max'). That
can't be changed. Capturing the host CPU type is a good idea.

> 
> Also, I've used the same CPU model that has been used on
> boot_linux_console.py:BootLinuxConsole.test_aarch64_virt, which may be
> a plus.

Is that the cortex-a53? Was some justification given for its use there?
If not, then maybe it should be changed there too.

Thanks,
drew