qemu-devel
[Top][All Lists]
Advanced
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [PATCH v2 00/23] Convert avocado tests to normal Python unittests

From: Richard Henderson
Subject: Re: [PATCH v2 00/23] Convert avocado tests to normal Python unittests
Date: Thu, 25 Jul 2024 20:42:31 +1000
User-agent: Mozilla Thunderbird 
On 7/25/24 19:55, Daniel P. Berrangé wrote:

On Thu, Jul 25, 2024 at 09:35:22AM +1000, Richard Henderson wrote:

On 7/25/24 03:52, Thomas Huth wrote:

The Avocado v88 that we use in QEMU is already on a life support
system: It is not supported by upstream anymore, and with the latest
versions of Python, it won't work anymore since it depends on the
"imp" module that has been removed in Python 3.12.

There have been several attempts to update the test suite in QEMU
to a newer version of Avocado, but so far no attempt has successfully
been merged yet.

Additionally, the whole "make check" test suite in QEMU is using the
meson test runner nowadays, so running the python-based tests via the
Avocodo test runner looks and feels quite like an oddball, requiring
the users to deal with the knowledge of multiple test runners in
parallel (e.g. the timeout settings work completely differently).

So instead of trying to update the python-based test suite in QEMU
to a newer version of Avocado, we should try to better integrate
it with the meson test runner instead. Indeed most tests work quite
nicely without the Avocado framework already, as you can see with
this patch series - it does not convert all tests, just a subset so
far, but this already proves that many tests only need small modifi-
cations to work without Avocado.

Only tests that use the LinuxTest / LinuxDistro and LinuxSSHMixIn
classes (e.g. based on cloud-init images or using SSH) really depend
on the Avocado framework, so we'd need a solution for those if we
want to continue using them. One solution might be to simply use the
required functions from avocado.utils for these tests, and still run
them via the meson test runner instead, but that needs some further
investigation that will be done later.


Now if you want to try out these patches: Apply the patches, then
recompile and then run:

   make check-functional

You can also run single targets e.g. with:

   make check-functional-ppc

You can also run the tests without any test runner now by
setting the PYTHONPATH environment variable to the "python" folder
of your source tree, and by specifying the build directory via
QEMU_BUILD_ROOT (if autodetection fails) and by specifying the
QEMU binary via QEMU_TEST_QEMU_BINARY. For example:

   export PYTHONPATH=$HOME/qemu/python
   export QEMU_TEST_QEMU_BINARY=qemu-system-x86_64
   export QEMU_BUILD_ROOT=$HOME/qemu/build
   ~/qemu/tests/functional/test_virtio_version.py

The logs of the tests can be found in the build directory under
tests/functional/<arch>/<testname> - console log and general logs will
be put in separate files there.

Still to be done: Update the documentation for this new test framework.


I'll say again that the download *must* be handled separately from the test
with timeout. This is an absolute show-stopper.

I've tried this twice now, from a decently fast connection in central
Brisbane, and have had multiple downloads be canceled by the timeout.  Since
the download isn't clever enough to pick up where it left off, it will never
succeed.


This is a tricky problem the way the tests are currently written, given the
desire for a minimal-change from the old avocado impl.

IIUC, avocado already had a per-test timeout, so would suffer the same
problem with downloads exploding the "normal" running time when cached.
Avocado runs a first pass doing all of the downloads, and only afterward runs the actual timed tests. I don't know the specifics of how, but it certainly obvious in the logging. 


Consider if we declared all required assets as class level variable

   class LinuxInitrd(QemuSystemTest):

       ASSETS = {
          "fedora18": {
            "url": ('https://archives.fedoraproject.org/pub/archive/fedora/li'
                      
'nux/releases/18/Fedora/x86_64/os/images/pxeboot/vmlinuz'),
             "hash": "'41464f68efe42b9991250bed86c7081d2ccdbb21'"
         }
       }

Then, we change the 'fetch_asset' method to take an asset name, not a
URL+hash:

       def test_with_2gib_file_should_exit_error_msg_with_linux_v3_6(self):
           kernel_path = self.fetch_asset("fedora18")

Now, 'fetch_asset' would lookup the URL + hash in the self.__class__.ASSETS
dict, so the test would run exactly as before.


Sure, that's one possibility.


This is all a non-trivial amount of work though, so I don't think
it is reasonable todo this as part of the immediate conversion in
this series.
I think that if we *don't* do this, then we cannot run in CI *at all* because we will be plagued with false timeouts.
And, frankly, any developer more than a few time zones away from the hosting of the asset will be continually frustrated. 

I repeat: this is a show-stopper.


r~
reply via email to
[Prev in Thread] Current Thread [Next in Thread]