[Qemu-devel] Functional tests (AKA Avocado-based tests)

[Cleber Rosa]

TL;DR
=====

This is about how QEMU developers can get started with functional
tests that are built on top of the Avocado libraries (and meant to be
run with the Avocado test runner).

The past
========

The Avocado project[1] has been working, for quite some time now, on a
"set of tools and libraries" with the goal of making writing tests
easier.  It is supposed to be a framework agnostic to the exact
software that will be under test.

But, at the same time, the Avocado project cannot deny its inheritance
and influences.  Those come from Autotest[2], which had "KVM Autotest"
as its largest and most developed "test".  This large Autotest test
(KVM Autotest) became virt-test[3] and later got integrated into
Avocado and became Avocado-VT[4] which is quite relevant here,
together with its QEMU test provider[5].

Avocado-VT and the QEMU test provider attempt to provide coverage
across platform and QEMU versions, which increases its complexity.
Also, it's built on a legacy set of principles and tools that makes
some developers stir away from it.

What's new?
===========

A few months ago, the Avocado developers returned to its
"virtualization origins", in an attempt to learn from the QEMU
project, and try to help with a way to have more functional tests in
the upstream QEMU repo.

We believe it's possible to expand the test coverage for QEMU by
facilitating
the creation of more functional tests QEMU.  This is no different than how
other types of tests are already included in the tree itself.

How
===

How we did it (so far)
----------------------

We're aware that there's a dilemma here: to be able to easily write
more powerful tests, a lot of the complexity has to be moved
elsewhere.  Here, it means moving complexity from the test itself to a
framework.  The QEMU source tree itself has proofs of this approach,
being the "scripts" and "tests/qemu-iotests" some of the examples.

Avocado itself[1] provides a lot of the code that should help to
absorb some of the complexities in writing tests, but not exactly
everything that is needed for QEMU.  The approach we believe will have
the best balance is to reuse upstream Avocado libraries whenever they
are useful and generic enough, and on top of that, libraries that are
part of QEMU itself.

How can you get started with it
-------------------------------

First of all, get Avocado installed.  Besides the Avocado test runner
itself, this will give you the basic libraries on which the other part
of this work was built on.  We want that to be simple and painless, so
here's our best bet for a one-liner installation:

  pip install --user avocado-framework
avocado-framework-plugin-varianter-yaml-to-mux aexpect

That will install Avocado within the user's home directory.  If you
give up on it, it can be uninstalled with another simple one-liner:

  pip uninstall -y avocado-framework
avocado-framework-plugin-varianter-yaml-to-mux aexpect

Now, suppose you're working on a given feature, and want to try your
luck writing a test using this work.  To avoid having you fetching and
rebasing from our currently in development fork[6] and branch[7], you
can just
add one commit to your tree with:

  curl
https://patch-diff.githubusercontent.com/raw/apahim/qemu/pull/17.patch |
git am -

This will get a simple patch from a snapshot branch[8].  You can, of course,
do it "the git way", fetching from that repo[6] and using the
non-snapshotted branch.

After that, we'd love for you to take a look at some of the existing
tests[9][10] and then attempt to create test for your own use case.
The basic README[11] file, and the Avocado documentation[12] are also
important resources not to be missed.

What's next?
============

Initially, feedback is what we're looking for.  It would be greatly
appreciated to understand if/how this suits (or not) use cases out
there.

After feedback, further refinements, and more tests are written, the
Avocado developers will follow up with an initial patch series for
upstream QEMU.  In such a proposal, we intend to have further
integration.  Ideally in way that "configure" can be given a
"--with-functional-[avocado-]tests" parameter of sorts, and a "make
 [functional-]check" would seamlessly include them.

Thanks!

References
==========

 [1] http://avocado-framework.github.io/
 [2] http://autotest.github.io/
 [3] https://github.com/autotest/virt-test
 [4] https://github.com/avocado-framework/avocado-vt
 [5] https://github.com/autotest/tp-qemu
 [6] https://github.com/apahim/qemu
 [7] https://github.com/apahim/qemu/tree/avocado_qemu
 [8] https://github.com/apahim/qemu/tree/avocado_qemu_snapshot
 [9]
https://github.com/apahim/qemu/blob/avocado_qemu/tests/avocado/test_info_memdev_host_nodes.py
[10]
https://github.com/apahim/qemu/blob/avocado_qemu/tests/avocado/test_ovmf_with_240_vcpus.py
[11]
https://github.com/apahim/qemu/blob/avocado_qemu/tests/avocado/README.rst
[12] http://avocado-framework.readthedocs.io/

-- 
Cleber Rosa
[ Sr Software Engineer - Virtualization Team - Red Hat ]
[ Avocado Test Framework - avocado-framework.github.io ]
[  7ABB 96EB 8B46 B94D 5E0F  E9BB 657E 8D33 A5F2 09F3  ]