Re: [PATCH v1 00/11] Convert avocado tests to normal Python unittests

[Cleber Rosa]

On Tue, Jul 16, 2024 at 7:28 AM Thomas Huth <thuth@redhat.com> wrote:
>
> The Avocado v88 that we use in QEMU is already on a life support
> system: It is not supported by upstream anymore, and with the latest
> versions of Python, it won't work anymore since it depends on the
> "imp" module that has been removed in Python 3.12.
>

Hi Thomas,

Let me start by saying that this is my attempt to explain the context
of the lack of updates on QEMU when it comes to the Avocado update,
and point to the actual updates.

> There have been several attempts to update the test suite in QEMU
> to a newer version of Avocado, but so far no attempt has successfully
> been merged yet.
>

So, we've seen in the past an attempt to update Avocado from 88.1 to a
regular release, and the troubles it caused, including a revert.  My
take was that a LTS version should be used, but during this time,
Avocado experienced a rewrite and having it replacing the old
implementation in a production level project such as QEMU was tricky.
Then 103.0 LTS was released, and there was extensive work to test the
QEMU tests before that release was cut.  Additionally, there was
further work, but unfortunately not posted yet, to make use of 103.0
features in the existing tests[2].   I've tested on GitLab with tests
running in parallel, cutting job times in 1/3[2].  A side node is
that, because 103.0 is an LTS release, it will receive the needed bug
fixes and updates that we deem necessary, including things we find in
QEMU tests.  In fact, 103.1[3] is in the works.

> Additionally, the whole "make check" test suite in QEMU is using the
> meson test runner nowadays, so running the python-based tests via the
> Avocodo test runner looks and feels quite like an oddball, requiring
> the users to deal with the knowledge of multiple test runners in
> parallel (e.g. the timeout settings work completely differently).
>

Now I believe we can be very much in sync here.  I've thought for a
while that there's no reason for Avocado to cooperate or be compatible
with Meson.  There's no reason why users can't simply pick how the
test gets run.  In fact, with the new Avocado architecture, you don't
even need to run "avocado" to run an "avocado-instrumented" test.  You
could pretty much run "avocado-runner-avocado-instrumented" with the
right parameters through Meson.

> So instead of trying to update the python-based test suite in QEMU
> to a newer version of Avocado, we should maybe try to better integrate
> it with the meson test runner instead. Indeed most tests work quite
> nicely without the Avocado framework already, as you can see with
> this patch series - it does not convert all tests, just a subset so
> far, but this already proves that many tests only need small modifi-
> cations to work without Avocado.
>
> Only tests that use the LinuxTest / LinuxDistro and LinuxSSHMixIn
> classes (e.g. based on cloud-init images or using SSH) really depend
> on the Avocado framework, so we'd need a solution for those if we
> want to continue using them. One solution might be to simply use the
> required functions from avocado.utils for these tests, and still run
> them via the meson test runner instead, but that needs some further
> investigation that will be done later.
>

So, I believe this type of higher level testing is something that
needs to remain, and even grow.  Speaking for Red Hat, I see the
movement of QE contributing more Avocado-VT style tests into QEMU
itself.  This means way more libraries and features that go into a
common set of utilities and features (more on that later) than it
currently exists in avocado.utils.

This brings the autils[4] initiative into the picture.  We're about
80% done with the project structure, and after that, it will be a
common utility project (such as the cloudinit and ssh) which can be
released automatically when the maintainer votes (through GitHub) that
a new release is needed.  With that, we can have all the existing QEMU
tests (and more) running in either meson or Avocado (for users that
want Avocado runner only features, such as dependency resolution,
container support, etc).

Now, if you think any of this makes sense, I'd love to cooperate with
you on an update that does both: update Avocado, and allow for the
execution of all tests through meson.

Thanks and I'm very very sorry for the delay in this update.
- Cleber.

[1] 
https://gitlab.com/cleber.gnu/qemu/-/commits/WIP/avocado_test_updates_lts?ref_type=heads
[2] https://gitlab.com/cleber.gnu/qemu/-/jobs/7085879478
[3] https://github.com/avocado-framework/avocado/pull/5953
[4] https://github.com/avocado-framework/autils