Re: [PATCH v1 00/11] Convert avocado tests to normal Python unittests

[Cleber Rosa]

On Mon, Jul 29, 2024 at 8:44 AM Daniel P. Berrangé <berrange@redhat.com> wrote:
>
> On Thu, Jul 25, 2024 at 10:21:54AM -0400, Cleber Rosa wrote:
> > On Tue, Jul 16, 2024 at 7:28 AM Thomas Huth <thuth@redhat.com> wrote:
> > > There have been several attempts to update the test suite in QEMU
> > > to a newer version of Avocado, but so far no attempt has successfully
> > > been merged yet.
> > >
> >
> > So, we've seen in the past an attempt to update Avocado from 88.1 to a
> > regular release, and the troubles it caused, including a revert.  My
> > take was that a LTS version should be used, but during this time,
> > Avocado experienced a rewrite and having it replacing the old
> > implementation in a production level project such as QEMU was tricky.
>
> IMHO the problems with updating avocado in QEMU are a sign that we're
> suffering from excess complexity, as it shouldn't be this difficult to
> update the test harness. That we need to stick on an LTS release to
> protect us from further instability further suggests that we would
> benefit from a much simpler solution here.
>

Hi Daniel,

Some other projects using Avocado have picked different, non-LTS
versions, to use as their pinned versions.  The consequences of using
rolling releases versus LTS-like releases is not something specific to
one software package IMO.  The suggestion to use LTS was to minimize
the impact on users writing and running tests, while allowing for only
needed fixes or very important features (say supporting a new Python
version) to be addressed.

> > > Additionally, the whole "make check" test suite in QEMU is using the
> > > meson test runner nowadays, so running the python-based tests via the
> > > Avocodo test runner looks and feels quite like an oddball, requiring
> > > the users to deal with the knowledge of multiple test runners in
> > > parallel (e.g. the timeout settings work completely differently).
> > >
> >
> > Now I believe we can be very much in sync here.  I've thought for a
> > while that there's no reason for Avocado to cooperate or be compatible
> > with Meson.  There's no reason why users can't simply pick how the
> > test gets run.  In fact, with the new Avocado architecture, you don't
> > even need to run "avocado" to run an "avocado-instrumented" test.  You
> > could pretty much run "avocado-runner-avocado-instrumented" with the
> > right parameters through Meson.
>
> Looking back to when we introduced avocado, we still had our original
> home grown build system, and multiple different ways to run integrate
> testing betwen our unit tests / qtests and block iotests. Adding
> avocado to the mix introduced a third way to run tests. All round it
> was rather a mess of inconsistent approaches to testing, but that was
> a fact of life with our old home grown build/test process.
>

Agreed.

> Today we have adopted a much more standardized build system via meson.
> While we do still have the block iotests harness, we now expose every
> test to meson individually to reduce what the intermediate harness
> needs to be involved in doing. Thomas' series here is bringing the same
> simplified integration to our functional tests, such that each functional
> test is known to meson individually. Overall we'll now have a consistent
> approach to integrating tests into meson, without intermediate harnesses
> adding their own logic.
>
> I think this will make it easier for QEMU maintainers to understand how
> the tests are working, and make it easier to maintain them and troubleshoot
> failures.
>

I think this is a very good way forward.  I don't think anything
(simple tests here) should depend on what they don't need to depend
on.

> So overall I'm not seeing a compelling reason for us to offer users a
> choice between meson & avocado how they run tests. Such choice is a maint
> burden over the long term, as we need to validate that both options keep
> working. Focusing on a single option reduces the complexity and maint
> work and gives us more flexiblity to evolve the code over time.
>
>

If meson is capable of doing all that can be done right now, then I
believe you're right.  No reason to use duplicate software.  Now, If
you mean that having "single option" may imply duplicating code that
already exists and is maintained elsewhere, into the QEMU repo, then
you're wrong.  This can be easily judged by how easy it is to convert
all current tests and their features (timeout, download management,
etc).

Another level of judgement that I believe would be helpful to the
decision making process here is how this impacts growing the test
coverage.  Tests that manipulate both QEMU and the guest are still
very very simple compared to what can be found on what I'll call "real
life integration testing" of QEMU (read Avocado-VT like tests). But
they present a glimpse of the problem. Having such tests requires a
good amount of code to either be used from a project/repo that
addresses that specifically, or be duplicated inside QEMU.

> > > So instead of trying to update the python-based test suite in QEMU
> > > to a newer version of Avocado, we should maybe try to better integrate
> > > it with the meson test runner instead. Indeed most tests work quite
> > > nicely without the Avocado framework already, as you can see with
> > > this patch series - it does not convert all tests, just a subset so
> > > far, but this already proves that many tests only need small modifi-
> > > cations to work without Avocado.
> > >
> > > Only tests that use the LinuxTest / LinuxDistro and LinuxSSHMixIn
> > > classes (e.g. based on cloud-init images or using SSH) really depend
> > > on the Avocado framework, so we'd need a solution for those if we
> > > want to continue using them. One solution might be to simply use the
> > > required functions from avocado.utils for these tests, and still run
> > > them via the meson test runner instead, but that needs some further
> > > investigation that will be done later.
> > >
> >
> > So, I believe this type of higher level testing is something that
> > needs to remain, and even grow.  Speaking for Red Hat, I see the
> > movement of QE contributing more Avocado-VT style tests into QEMU
> > itself.  This means way more libraries and features that go into a
> > common set of utilities and features (more on that later) than it
> > currently exists in avocado.utils.
>
> The series here is showing how simple it is to adapt our existing
> tests to the new execution approach, so shouldn't have a significant
> impact on ability of people to contribute futher tests in future.
>

I am on the side of the unified execution approach.  But, will it
really be simple to convert all tests?  "Real life integration" tests
too?  I would love to see that, but I can't see how that can be done
without the woes mentioned before (duplicating other code or limiting
the further toolset and capabilities of test writers).

Best regards,
- Cleber.