Re: [RFC PATCH 0/8] Convert avocado tests to normal Python unittests

[Daniel P . Berrangé]

On Thu, Jul 11, 2024 at 01:55:38PM +0200, Thomas Huth wrote:
> The Avocado v88 that we use in QEMU is already on a life support
> system: It is not supported by upstream anymore, and with the latest
> versions of Python, it won't work anymore since it depends on the
> "imp" module that has been removed in Python 3.12.
> 
> There have been several attempts to update the test suite in QEMU
> to a newer version of Avocado, but so far no attempt has successfully
> been merged yet.
> 
> Additionally, the whole "make check" test suite in QEMU is using the
> meson test runner nowadays, so running the python-based tests via the
> Avocodo test runner looks and feels quite like an oddball, requiring
> the users to deal with the knowledge of multiple test runners in
> parallel.

The fewer / simpler the layers we have in the execution path
of tests the better our life will be in debugging IMHO.

Having each individual test registered with meson has the
particularly strong advantage that we can make use of meson's
timeout feature to force individual tests to abort if they
hang/run too slowly, as we did when converting the iotests
to individual meson tests.

> 
> So instead of trying to update the python-based test suite in QEMU
> to a newer version of Avocado, we should maybe try to better integrate
> it with the meson test runner instead. Indeed most tests work quite
> nicely without the Avocado framework already, as you can see with
> this patch series - it does not convert all tests, just a subset since
> it is just an RFC so far, but as you can see, many tests only need
> small modifications to work without Avocado.
> 
> If you want to try it: Apply the patches, make sure that you have the
> "pytest" program installed, then recompile and then run:
> 
>  make check-pytest
> 
> Things that need further attention though:
> 
> - All tests that use the LinuxTest / LinuxDistro classes (e.g. based
>   on cloud-init images) really depend on the Avocado framework,
>   thus we'd need a solution for those if we want to continue with
>   this approach

Right, avocado is providing 2 distinct things, the test execution
harness and the test framework APIs.

It could be valid to remove use of the harness but keep using
the framework APIs, especially if that's sufficient to unblock
updating to new avocado versions too ? Over the longer term
we can consider whether the framework APIs should remain or
be replaced by something else.

> - Same for all tests that require the LinuxSSHMixIn class - we'd
>   need to provide a solution for ssh-based tests, too.
> 
> - We lose the way of running tests via the avocado tags this way...
>   single targets can still be tested by running "make check-pytest-arm"
>   for example, but running selected tests by other tags does not
>   work anymore.

The meson "suites" concept is the logical equivalent of tags.
You've wired up a suite for each architecture. We could define
more suites if there are other useful criteria for filtering
tests to be run. Perhaps machine type ? "make check-pytest-arm-<machine>"

> - I haven't looked into logging yet ... this still needs some work
>   so that you could e.g. inspect the console output of the guests
>   somewhere

Yep, debuggability is probably the single biggest problem we face
with our tests. Simplifying the test execution harness will help
in this respect, but yeah, we must have a way to capture logs
of stuff executed.

> - I did not work on documentation updates yet (will do that if we
>   agree to continue with this patch series)
> 
> What's your thoughts? Is it worth to continue with this approach?
> Or shall I rather forget about it and wait for the Avocado version
> update?



With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|