Re: [Qemu-devel] Functional tests (AKA Avocado-based tests)

[Alistair Francis]

On Mon, Feb 5, 2018 at 8:34 AM, Cleber Rosa <address@hidden> wrote:
>
>
> On 02/01/2018 07:10 PM, Alistair Francis wrote:
>> On Wed, Jan 17, 2018 at 4:47 PM, Cleber Rosa <address@hidden> wrote:
>>>
>>>
>>> On 01/17/2018 06:41 PM, Alistair Francis wrote:
>>>> On Wed, Jan 17, 2018 at 12:05 AM, Cleber Rosa <address@hidden> wrote:
>>>>> TL;DR
>>>>> =====
>>>>>
>>>>> This is about how QEMU developers can get started with functional
>>>>> tests that are built on top of the Avocado libraries (and meant to be
>>>>> run with the Avocado test runner).
>>>>>
>>>>> The past
>>>>> ========
>>>>>
>>>>> The Avocado project[1] has been working, for quite some time now, on a
>>>>> "set of tools and libraries" with the goal of making writing tests
>>>>> easier.  It is supposed to be a framework agnostic to the exact
>>>>> software that will be under test.
>>>>>
>>>>> But, at the same time, the Avocado project cannot deny its inheritance
>>>>> and influences.  Those come from Autotest[2], which had "KVM Autotest"
>>>>> as its largest and most developed "test".  This large Autotest test
>>>>> (KVM Autotest) became virt-test[3] and later got integrated into
>>>>> Avocado and became Avocado-VT[4] which is quite relevant here,
>>>>> together with its QEMU test provider[5].
>>>>>
>>>>> Avocado-VT and the QEMU test provider attempt to provide coverage
>>>>> across platform and QEMU versions, which increases its complexity.
>>>>> Also, it's built on a legacy set of principles and tools that makes
>>>>> some developers stir away from it.
>>>>>
>>>>> What's new?
>>>>> ===========
>>>>>
>>>>> A few months ago, the Avocado developers returned to its
>>>>> "virtualization origins", in an attempt to learn from the QEMU
>>>>> project, and try to help with a way to have more functional tests in
>>>>> the upstream QEMU repo.
>>>>>
>>>>> We believe it's possible to expand the test coverage for QEMU by
>>>>> facilitating
>>>>> the creation of more functional tests QEMU.  This is no different than how
>>>>> other types of tests are already included in the tree itself.
>>>>>
>>>>> How
>>>>> ===
>>>>>
>>>>> How we did it (so far)
>>>>> ----------------------
>>>>>
>>>>> We're aware that there's a dilemma here: to be able to easily write
>>>>> more powerful tests, a lot of the complexity has to be moved
>>>>> elsewhere.  Here, it means moving complexity from the test itself to a
>>>>> framework.  The QEMU source tree itself has proofs of this approach,
>>>>> being the "scripts" and "tests/qemu-iotests" some of the examples.
>>>>>
>>>>> Avocado itself[1] provides a lot of the code that should help to
>>>>> absorb some of the complexities in writing tests, but not exactly
>>>>> everything that is needed for QEMU.  The approach we believe will have
>>>>> the best balance is to reuse upstream Avocado libraries whenever they
>>>>> are useful and generic enough, and on top of that, libraries that are
>>>>> part of QEMU itself.
>>>>>
>>>>> How can you get started with it
>>>>> -------------------------------
>>>>>
>>>>> First of all, get Avocado installed.  Besides the Avocado test runner
>>>>> itself, this will give you the basic libraries on which the other part
>>>>> of this work was built on.  We want that to be simple and painless, so
>>>>> here's our best bet for a one-liner installation:
>>>>>
>>>>>   pip install --user avocado-framework
>>>>> avocado-framework-plugin-varianter-yaml-to-mux aexpect
>>>>>
>>>>> That will install Avocado within the user's home directory.  If you
>>>>> give up on it, it can be uninstalled with another simple one-liner:
>>>>>
>>>>>   pip uninstall -y avocado-framework
>>>>> avocado-framework-plugin-varianter-yaml-to-mux aexpect
>>>>>
>>>>> Now, suppose you're working on a given feature, and want to try your
>>>>> luck writing a test using this work.  To avoid having you fetching and
>>>>> rebasing from our currently in development fork[6] and branch[7], you
>>>>> can just
>>>>> add one commit to your tree with:
>>>>>
>>>>>   curl
>>>>> https://patch-diff.githubusercontent.com/raw/apahim/qemu/pull/17.patch |
>>>>> git am -
>>>>>
>>>>> This will get a simple patch from a snapshot branch[8].  You can, of 
>>>>> course,
>>>>> do it "the git way", fetching from that repo[6] and using the
>>>>> non-snapshotted branch.
>>>>>
>>>>> After that, we'd love for you to take a look at some of the existing
>>>>> tests[9][10] and then attempt to create test for your own use case.
>>>>> The basic README[11] file, and the Avocado documentation[12] are also
>>>>> important resources not to be missed.
>>>>>
>>>>> What's next?
>>>>> ============
>>>>>
>>>>> Initially, feedback is what we're looking for.  It would be greatly
>>>>> appreciated to understand if/how this suits (or not) use cases out
>>>>> there.
>>>>>
>>>>> After feedback, further refinements, and more tests are written, the
>>>>> Avocado developers will follow up with an initial patch series for
>>>>> upstream QEMU.  In such a proposal, we intend to have further
>>>>> integration.  Ideally in way that "configure" can be given a
>>>>> "--with-functional-[avocado-]tests" parameter of sorts, and a "make
>>>>>  [functional-]check" would seamlessly include them.
>>>>
>>>> I have a few thoughts.
>>>>
>>>> We use pytest/pexpect internally to kick off QEMU runs and monitor the
>>>> output (no interaction with the QEMU source tree) and I think it is
>>>> extremely useful. So I am all for using Python to test things and this
>>>> looks really well done!
>>>>
>>>
>>> Thanks for checking it out, and for the positive words.  Now, sorry if
>>> I'm missing some obvious information, but is this work of yours with
>>> pytest/pexpect publicly available?  I'd like to also take a look at
>>> that, because it does look similar to the Avocado + aexpect approach
>>> taken here.
>>
>> Unfortunately it's not and it would take months for us to be able to
>> make it available.
>>
>
> I see.
>
>>>
>>>> What I don't understand though is what this gives us compared to the
>>>> existing QEMU test infrastructure? Besides being able to use Python
>>>> and a better interface what are the main benefits? I think that is
>>>> something worth documenting somewhere.
>>>>
>>>
>>> We currently intend to *add* to the QEMU test infrastructure, not
>>> replace it.
>>
>> Is there a benefit of integrating it into the tree then? It's always
>> possible to have an out of tree testing framework.
>>
>
> Upstream first is just the general modus operandi that we have.  Then,
> we want to have as much testing as possible as early as possible.
>
> Finally, by having tests in-tree, they can be seen in different light.
> What I mean is that when tests are in-tree, they will (or should) really
> map to the current state of the code they test (per commit).  If a
> feature changes behavior upstream, the respective test will also need a
> change at the same time, and as such, will mean an *intended* change and
> not a regression.  With out-of-tree tests, it's pretty hard to keep this
> synchronization and regressions will slip in, *at least* for some time.
>
> I guess the question here is actually the opposite: do you see any
> problems with in-tree tests?  Since you keep out of tree tests, I would
> honestly like to hear about your experience.

For us the main reason is that we want to avoid editing the fork as
much as possible, so we aren't going to add tests to our fork.

The other main advantage I see is that it allows the tests to be
updated more easily as the changes don't need to go through the more
rigorous QEMU review process.

You are correct though that synchronisation is an issue.

Alistair

>
>>>
>>> The benefits we envision are, besides hopefully easier and more capable
>>> interfaces, to simply have more upstream tests.  This means avoiding new
>>> regressions and improving coverage.
>>>
>>>> Also, it looks like this will require images checked into git
>>>> somewhere is that correct? Is there a good plan on how to handle that?
>>>>
>>>
>>> It won't require images checked into git.  Right now, tests use the
>>> vmimage library:
>>>
>>> http://avocado-framework.readthedocs.io/en/57.0/api/utils/avocado.utils.html#avocado.utils.vmimage.get
>>>
>>> Which downloads (and caches) images from external sources.
>>
>> Ah! That's cool. Managing images is one of the challenges we have at the 
>> moment.
>>
>> Alistair
>>
>
> Nice that you like it.  We also find a library such as "vmimage" is
> something simple enough, but still not quite something that would live
> in the QEMU tree.
>
> - Cleber.
>
>>>
>>> Please let me know if you have more questions!
>>> - Cleber.
>>>
>>>> Alistair
>>>>
>>>>>
>>>>> Thanks!
>>>>>
>>>>> References
>>>>> ==========
>>>>>
>>>>>  [1] http://avocado-framework.github.io/
>>>>>  [2] http://autotest.github.io/
>>>>>  [3] https://github.com/autotest/virt-test
>>>>>  [4] https://github.com/avocado-framework/avocado-vt
>>>>>  [5] https://github.com/autotest/tp-qemu
>>>>>  [6] https://github.com/apahim/qemu
>>>>>  [7] https://github.com/apahim/qemu/tree/avocado_qemu
>>>>>  [8] https://github.com/apahim/qemu/tree/avocado_qemu_snapshot
>>>>>  [9]
>>>>> https://github.com/apahim/qemu/blob/avocado_qemu/tests/avocado/test_info_memdev_host_nodes.py
>>>>> [10]
>>>>> https://github.com/apahim/qemu/blob/avocado_qemu/tests/avocado/test_ovmf_with_240_vcpus.py
>>>>> [11]
>>>>> https://github.com/apahim/qemu/blob/avocado_qemu/tests/avocado/README.rst
>>>>> [12] http://avocado-framework.readthedocs.io/
>>>>>
>>>>> --
>>>>> Cleber Rosa
>>>>> [ Sr Software Engineer - Virtualization Team - Red Hat ]
>>>>> [ Avocado Test Framework - avocado-framework.github.io ]
>>>>> [  7ABB 96EB 8B46 B94D 5E0F  E9BB 657E 8D33 A5F2 09F3  ]
>>>>>
>>>
>>> --
>>> Cleber Rosa
>>> [ Sr Software Engineer - Virtualization Team - Red Hat ]
>>> [ Avocado Test Framework - avocado-framework.github.io ]
>>> [  7ABB 96EB 8B46 B94D 5E0F  E9BB 657E 8D33 A5F2 09F3  ]
>
> --
> Cleber Rosa
> [ Sr Software Engineer - Virtualization Team - Red Hat ]
> [ Avocado Test Framework - avocado-framework.github.io ]
> [  7ABB 96EB 8B46 B94D 5E0F  E9BB 657E 8D33 A5F2 09F3  ]