[PATCH v1 00/11] Convert avocado tests to normal Python unittests

[Thomas Huth]

The Avocado v88 that we use in QEMU is already on a life support
system: It is not supported by upstream anymore, and with the latest
versions of Python, it won't work anymore since it depends on the
"imp" module that has been removed in Python 3.12.

There have been several attempts to update the test suite in QEMU
to a newer version of Avocado, but so far no attempt has successfully
been merged yet.

Additionally, the whole "make check" test suite in QEMU is using the
meson test runner nowadays, so running the python-based tests via the
Avocodo test runner looks and feels quite like an oddball, requiring
the users to deal with the knowledge of multiple test runners in
parallel (e.g. the timeout settings work completely differently).

So instead of trying to update the python-based test suite in QEMU
to a newer version of Avocado, we should maybe try to better integrate
it with the meson test runner instead. Indeed most tests work quite
nicely without the Avocado framework already, as you can see with
this patch series - it does not convert all tests, just a subset so
far, but this already proves that many tests only need small modifi-
cations to work without Avocado.

Only tests that use the LinuxTest / LinuxDistro and LinuxSSHMixIn
classes (e.g. based on cloud-init images or using SSH) really depend
on the Avocado framework, so we'd need a solution for those if we
want to continue using them. One solution might be to simply use the
required functions from avocado.utils for these tests, and still run
them via the meson test runner instead, but that needs some further
investigation that will be done later.


Now if you want to try out these patches: Apply the patches, then
recompile and then run:

 make check-functional

You can also run single targets e.g. with:

 make check-functional-ppc

You can also run the tests without any test runner now by
setting the PYTHONPATH environment variable to the "python" folder
of your source tree, and by specifying the build directory via
QEMU_BUILD_ROOT (if autodetection fails) and by specifying the
QEMU binary via QEMU_TEST_QEMU_BINARY. For example:

 export PYTHONPATH=$HOME/qemu/python
 export QEMU_TEST_QEMU_BINARY=qemu-system-x86_64
 export PYTHONPATH=$HOME/qemu/build
 ~/qemu/tests/functional/test_virtio_version.py

The logs of the tests can be found in the build directory under
tests/functional/<testname> - console log and general logs will
be put in separate files there.

Still to be done: Update the documentation for this new test framework.

RFC -> v1:
- Now using pycotap for running the tests instead of "pytest"
- Change the name from "tests/pytest" to "tests/functional" accordingly
- Make it possible to run the tests directly
- Use Python's urllib instead of wget for downloading
- Lots of makefile / meson integration improvements
- Converted more tests
- Update MAINTAINERS file accordingly
- Added a patch to run check-functional in the gitlab-CI
- ... lots of other changes I forgot about ... in fact, I changed so
  many things that I also did not dare to pick up the Reviewed-bys
  from the RFC

Thomas Huth (11):
  tests/functional: Add base classes for the upcoming pytest-based tests
  tests/functional: Convert simple avocado tests into standalone python
    tests
  tests/functional: Convert avocado tests that just need a small
    adjustment
  tests/functional: Add python-based tests to the meson build system
  tests/functional: Implement fetch_asset() method for downloading
    assets
  tests/functional: Convert some tests that download files via
    fetch_asset()
  tests/functional: Add a function for extracting files from an archive
  tests/functional: Convert some avocado tests that needed
    avocado.utils.archive
  tests/functional: Set up logging
  tests/functional: Convert the s390x avocado tests into standalone
    tests
  gitlab-ci: Add "check-functional" to the build tests

 MAINTAINERS                                   |  22 +-
 .gitlab-ci.d/buildtest-template.yml           |   3 +-
 .gitlab-ci.d/buildtest.yml                    |  60 +--
 pythondeps.toml                               |   3 +-
 tests/Makefile.include                        |  18 +-
 tests/functional/meson.build                  | 112 +++++
 tests/functional/qemu_test/__init__.py        | 384 ++++++++++++++++++
 tests/functional/qemu_test/utils.py           |  28 ++
 .../test_arm_canona1100.py}                   |  21 +-
 .../test_arm_n8x0.py}                         |  25 +-
 .../test_avr_mega2560.py}                     |  11 +-
 .../test_cpu_queries.py}                      |   7 +-
 .../test_empty_cpu_model.py}                  |   7 +-
 .../test_info_usernet.py}                     |  11 +-
 .../test_loongarch64_virt.py}                 |  16 +-
 .../test_mem_addr_space.py}                   |  53 +--
 .../test_mips64el_loongson3v.py}              |  26 +-
 .../test_netdev_ethtool.py}                   |  32 +-
 .../test_pc_cpu_hotplug_props.py}             |  11 +-
 .../ppc_405.py => functional/test_ppc_405.py} |  19 +-
 .../test_ppc_74xx.py}                         |  74 ++--
 .../test_ppc_bamboo.py}                       |  23 +-
 .../test_s390x_ccw_virtio.py}                 |  32 +-
 .../test_s390x_topology.py}                   |  70 ++--
 .../test_sparc64_sun4u.py}                    |  25 +-
 .../version.py => functional/test_version.py} |  13 +-
 .../test_virtio_version.py}                   |   8 +-
 tests/meson.build                             |   1 +
 28 files changed, 795 insertions(+), 320 deletions(-)
 create mode 100644 tests/functional/meson.build
 create mode 100644 tests/functional/qemu_test/__init__.py
 create mode 100644 tests/functional/qemu_test/utils.py
 rename tests/{avocado/machine_arm_canona1100.py => 
functional/test_arm_canona1100.py} (71%)
 mode change 100644 => 100755
 rename tests/{avocado/machine_arm_n8x0.py => functional/test_arm_n8x0.py} (71%)
 mode change 100644 => 100755
 rename tests/{avocado/machine_avr6.py => functional/test_avr_mega2560.py} (90%)
 mode change 100644 => 100755
 rename tests/{avocado/cpu_queries.py => functional/test_cpu_queries.py} (89%)
 mode change 100644 => 100755
 rename tests/{avocado/empty_cpu_model.py => 
functional/test_empty_cpu_model.py} (84%)
 mode change 100644 => 100755
 rename tests/{avocado/info_usernet.py => functional/test_info_usernet.py} (87%)
 mode change 100644 => 100755
 rename tests/{avocado/machine_loongarch.py => 
functional/test_loongarch64_virt.py} (89%)
 mode change 100644 => 100755
 rename tests/{avocado/mem-addr-space-check.py => 
functional/test_mem_addr_space.py} (92%)
 mode change 100644 => 100755
 rename tests/{avocado/machine_mips_loongson3v.py => 
functional/test_mips64el_loongson3v.py} (55%)
 mode change 100644 => 100755
 rename tests/{avocado/netdev-ethtool.py => functional/test_netdev_ethtool.py} 
(81%)
 mode change 100644 => 100755
 rename tests/{avocado/pc_cpu_hotplug_props.py => 
functional/test_pc_cpu_hotplug_props.py} (90%)
 mode change 100644 => 100755
 rename tests/{avocado/ppc_405.py => functional/test_ppc_405.py} (73%)
 mode change 100644 => 100755
 rename tests/{avocado/ppc_74xx.py => functional/test_ppc_74xx.py} (74%)
 mode change 100644 => 100755
 rename tests/{avocado/ppc_bamboo.py => functional/test_ppc_bamboo.py} (75%)
 mode change 100644 => 100755
 rename tests/{avocado/machine_s390_ccw_virtio.py => 
functional/test_s390x_ccw_virtio.py} (95%)
 mode change 100644 => 100755
 rename tests/{avocado/s390_topology.py => functional/test_s390x_topology.py} 
(90%)
 mode change 100644 => 100755
 rename tests/{avocado/machine_sparc64_sun4u.py => 
functional/test_sparc64_sun4u.py} (60%)
 mode change 100644 => 100755
 rename tests/{avocado/version.py => functional/test_version.py} (78%)
 mode change 100644 => 100755
 rename tests/{avocado/virtio_version.py => functional/test_virtio_version.py} 
(98%)
 mode change 100644 => 100755

-- 
2.45.2