Re: [Qemu-devel] Functional tests (AKA Avocado-based tests)

[Alistair Francis]

On Wed, Jan 17, 2018 at 12:05 AM, Cleber Rosa <address@hidden> wrote:
> TL;DR
> =====
>
> This is about how QEMU developers can get started with functional
> tests that are built on top of the Avocado libraries (and meant to be
> run with the Avocado test runner).
>
> The past
> ========
>
> The Avocado project[1] has been working, for quite some time now, on a
> "set of tools and libraries" with the goal of making writing tests
> easier.  It is supposed to be a framework agnostic to the exact
> software that will be under test.
>
> But, at the same time, the Avocado project cannot deny its inheritance
> and influences.  Those come from Autotest[2], which had "KVM Autotest"
> as its largest and most developed "test".  This large Autotest test
> (KVM Autotest) became virt-test[3] and later got integrated into
> Avocado and became Avocado-VT[4] which is quite relevant here,
> together with its QEMU test provider[5].
>
> Avocado-VT and the QEMU test provider attempt to provide coverage
> across platform and QEMU versions, which increases its complexity.
> Also, it's built on a legacy set of principles and tools that makes
> some developers stir away from it.
>
> What's new?
> ===========
>
> A few months ago, the Avocado developers returned to its
> "virtualization origins", in an attempt to learn from the QEMU
> project, and try to help with a way to have more functional tests in
> the upstream QEMU repo.
>
> We believe it's possible to expand the test coverage for QEMU by
> facilitating
> the creation of more functional tests QEMU.  This is no different than how
> other types of tests are already included in the tree itself.
>
> How
> ===
>
> How we did it (so far)
> ----------------------
>
> We're aware that there's a dilemma here: to be able to easily write
> more powerful tests, a lot of the complexity has to be moved
> elsewhere.  Here, it means moving complexity from the test itself to a
> framework.  The QEMU source tree itself has proofs of this approach,
> being the "scripts" and "tests/qemu-iotests" some of the examples.
>
> Avocado itself[1] provides a lot of the code that should help to
> absorb some of the complexities in writing tests, but not exactly
> everything that is needed for QEMU.  The approach we believe will have
> the best balance is to reuse upstream Avocado libraries whenever they
> are useful and generic enough, and on top of that, libraries that are
> part of QEMU itself.
>
> How can you get started with it
> -------------------------------
>
> First of all, get Avocado installed.  Besides the Avocado test runner
> itself, this will give you the basic libraries on which the other part
> of this work was built on.  We want that to be simple and painless, so
> here's our best bet for a one-liner installation:
>
>   pip install --user avocado-framework
> avocado-framework-plugin-varianter-yaml-to-mux aexpect
>
> That will install Avocado within the user's home directory.  If you
> give up on it, it can be uninstalled with another simple one-liner:
>
>   pip uninstall -y avocado-framework
> avocado-framework-plugin-varianter-yaml-to-mux aexpect
>
> Now, suppose you're working on a given feature, and want to try your
> luck writing a test using this work.  To avoid having you fetching and
> rebasing from our currently in development fork[6] and branch[7], you
> can just
> add one commit to your tree with:
>
>   curl
> https://patch-diff.githubusercontent.com/raw/apahim/qemu/pull/17.patch |
> git am -
>
> This will get a simple patch from a snapshot branch[8].  You can, of course,
> do it "the git way", fetching from that repo[6] and using the
> non-snapshotted branch.
>
> After that, we'd love for you to take a look at some of the existing
> tests[9][10] and then attempt to create test for your own use case.
> The basic README[11] file, and the Avocado documentation[12] are also
> important resources not to be missed.
>
> What's next?
> ============
>
> Initially, feedback is what we're looking for.  It would be greatly
> appreciated to understand if/how this suits (or not) use cases out
> there.
>
> After feedback, further refinements, and more tests are written, the
> Avocado developers will follow up with an initial patch series for
> upstream QEMU.  In such a proposal, we intend to have further
> integration.  Ideally in way that "configure" can be given a
> "--with-functional-[avocado-]tests" parameter of sorts, and a "make
>  [functional-]check" would seamlessly include them.

I have a few thoughts.

We use pytest/pexpect internally to kick off QEMU runs and monitor the
output (no interaction with the QEMU source tree) and I think it is
extremely useful. So I am all for using Python to test things and this
looks really well done!

What I don't understand though is what this gives us compared to the
existing QEMU test infrastructure? Besides being able to use Python
and a better interface what are the main benefits? I think that is
something worth documenting somewhere.

Also, it looks like this will require images checked into git
somewhere is that correct? Is there a good plan on how to handle that?

Alistair

>
> Thanks!
>
> References
> ==========
>
>  [1] http://avocado-framework.github.io/
>  [2] http://autotest.github.io/
>  [3] https://github.com/autotest/virt-test
>  [4] https://github.com/avocado-framework/avocado-vt
>  [5] https://github.com/autotest/tp-qemu
>  [6] https://github.com/apahim/qemu
>  [7] https://github.com/apahim/qemu/tree/avocado_qemu
>  [8] https://github.com/apahim/qemu/tree/avocado_qemu_snapshot
>  [9]
> https://github.com/apahim/qemu/blob/avocado_qemu/tests/avocado/test_info_memdev_host_nodes.py
> [10]
> https://github.com/apahim/qemu/blob/avocado_qemu/tests/avocado/test_ovmf_with_240_vcpus.py
> [11]
> https://github.com/apahim/qemu/blob/avocado_qemu/tests/avocado/README.rst
> [12] http://avocado-framework.readthedocs.io/
>
> --
> Cleber Rosa
> [ Sr Software Engineer - Virtualization Team - Red Hat ]
> [ Avocado Test Framework - avocado-framework.github.io ]
> [  7ABB 96EB 8B46 B94D 5E0F  E9BB 657E 8D33 A5F2 09F3  ]
>