The Avocado v88 that we use in QEMU is already on a life support
system: It is not supported by upstream anymore, and with the latest
versions of Python, it won't work anymore since it depends on the
"imp" module that has been removed in Python 3.12.
There have been several attempts to update the test suite in QEMU
to a newer version of Avocado, but so far no attempt has successfully
been merged yet.
Additionally, the whole "make check" test suite in QEMU is using the
meson test runner nowadays, so running the python-based tests via the
Avocodo test runner looks and feels quite like an oddball, requiring
the users to deal with the knowledge of multiple test runners in
parallel.
So instead of trying to update the python-based test suite in QEMU
to a newer version of Avocado, we should maybe try to better integrate
it with the meson test runner instead. Indeed most tests work quite
nicely without the Avocado framework already, as you can see with
this patch series - it does not convert all tests, just a subset since
it is just an RFC so far, but as you can see, many tests only need
small modifications to work without Avocado.
If you want to try it: Apply the patches, make sure that you have the
"pytest" program installed, then recompile and then run:
make check-pytest
Things that need further attention though:
- All tests that use the LinuxTest / LinuxDistro classes (e.g. based
on cloud-init images) really depend on the Avocado framework,
thus we'd need a solution for those if we want to continue with
this approach
- Same for all tests that require the LinuxSSHMixIn class - we'd
need to provide a solution for ssh-based tests, too.
- We lose the way of running tests via the avocado tags this way...
single targets can still be tested by running "make check-pytest-arm"
for example, but running selected tests by other tags does not
work anymore.
- I haven't looked into logging yet ... this still needs some work
so that you could e.g. inspect the console output of the guests
somewhere
This has spilled the most developer blood of any other problem with the Python-based tests. Be very careful here.
I still have a prototype for replacing QMPMachine with an asyncio variant that should have more robust logging features, but I put it on the back-burner.
Avocado tests are the primary user of the QMP Machine interface I hate the very most, a multi-threaded buffer-reader that works only by the grace of god. If you do go down this path, I may want to take the opportunity to abolish that interface once and for all.
I think simplifying the console buffering will help ease debuggability.
(Note, this isn't an avocado exclusive problem so much as it is the emergent evolution of both qmp machine and avocado developing their own solutions to console logging problems, resulting in two layers that are trying to do similar things.)
- I did not work on documentation updates yet (will do that if we
agree to continue with this patch series)
What's your thoughts? Is it worth to continue with this approach?
Or shall I rather forget about it and wait for the Avocado version
update?
I'm personally ambivalent on avocado; I use it for the python self-tests as dogfooding but I can likely switch back over to plain pytest if that's the direction we head. I don't think I use any crazy features except some asyncio helpers i advocated for. I'm not sure what pytest's asyncio support looks like, but I have to imagine as the premier testing framework that it has *something* for me to use.
My only ask is that we keep the tests running in the custom venv environment we set up at build time. We have some funky post-hoc initialization of avocado that allows us to use internet packages post-config for testing purposes. If we move to pytest, it's possible we can eliminate that funkiness, which would be a win.
I'm also not so sure about recreating all of the framework that pulls vm images on demand, that sounds like it'd be a lot of work, but maybe I'm wrong about that.
Tacit ACK from me on this project in general, provided we are still using the configure venv.
Thomas
Ani Sinha (1):
tests/pytest: add pytest to the meson build system
Thomas Huth (7):
tests/pytest: Add base classes for the upcoming pytest-based tests
tests/pytest: Convert some simple avocado tests into pytests
tests/pytest: Convert info_usernet and version test with small
adjustments
tests_pytest: Implement fetch_asset() method for downloading assets
tests/pytest: Convert some tests that download files via fetch_asset()
tests/pytest: Add a function for extracting files from an archive
tests/pytest: Convert avocado test that needed avocado.utils.archive
tests/Makefile.include | 4 +-
tests/meson.build | 1 +
tests/pytest/meson.build | 74 ++++
tests/pytest/qemu_pytest/__init__.py | 362 ++++++++++++++++++
tests/pytest/qemu_pytest/utils.py | 21 +
.../test_arm_canona1100.py} | 16 +-
.../test_cpu_queries.py} | 2 +-
.../test_empty_cpu_model.py} | 2 +-
.../test_info_usernet.py} | 6 +-
.../test_machine_arm_n8x0.py} | 20 +-
.../test_machine_avr6.py} | 7 +-
.../test_machine_loongarch.py} | 11 +-
.../test_machine_mips_loongson3v.py} | 19 +-
.../test_mem_addr_space.py} | 3 +-
.../test_ppc_bamboo.py} | 18 +-
.../version.py => pytest/test_version.py} | 8 +-
.../test_virtio_version.py} | 2 +-
17 files changed, 502 insertions(+), 74 deletions(-)
create mode 100644 tests/pytest/meson.build
create mode 100644 tests/pytest/qemu_pytest/__init__.py
create mode 100644 tests/pytest/qemu_pytest/utils.py
rename tests/{avocado/machine_arm_canona1100.py => pytest/test_arm_canona1100.py} (74%)
rename tests/{avocado/cpu_queries.py => pytest/test_cpu_queries.py} (96%)
rename tests/{avocado/empty_cpu_model.py => pytest/test_empty_cpu_model.py} (94%)
rename tests/{avocado/info_usernet.py => pytest/test_info_usernet.py} (91%)
rename tests/{avocado/machine_arm_n8x0.py => pytest/test_machine_arm_n8x0.py} (71%)
rename tests/{avocado/machine_avr6.py => pytest/test_machine_avr6.py} (91%)
rename tests/{avocado/machine_loongarch.py => pytest/test_machine_loongarch.py} (89%)
rename tests/{avocado/machine_mips_loongson3v.py => pytest/test_machine_mips_loongson3v.py} (59%)
rename tests/{avocado/mem-addr-space-check.py => pytest/test_mem_addr_space.py} (99%)
rename tests/{avocado/ppc_bamboo.py => pytest/test_ppc_bamboo.py} (75%)
rename tests/{avocado/version.py => pytest/test_version.py} (82%)
rename tests/{avocado/virtio_version.py => pytest/test_virtio_version.py} (99%)
--
2.45.2