Re: [Qemu-devel] Functional tests (AKA Avocado-based tests)

[Lukáš Doktor]

Dne 9.2.2018 v 00:41 Alistair Francis napsal(a):
> On Thu, Feb 8, 2018 at 3:38 PM, Alistair Francis <address@hidden> wrote:
>> On Mon, Feb 5, 2018 at 8:34 AM, Cleber Rosa <address@hidden> wrote:
>>>
>>>
>>> On 02/01/2018 07:10 PM, Alistair Francis wrote:
>>>> On Wed, Jan 17, 2018 at 4:47 PM, Cleber Rosa <address@hidden> wrote:
>>>>>
>>>>>
>>>>> On 01/17/2018 06:41 PM, Alistair Francis wrote:
>>>>>> On Wed, Jan 17, 2018 at 12:05 AM, Cleber Rosa <address@hidden> wrote:
>>>>>>> TL;DR
>>>>>>> =====
>>>>>>>
>>>>>>> This is about how QEMU developers can get started with functional
>>>>>>> tests that are built on top of the Avocado libraries (and meant to be
>>>>>>> run with the Avocado test runner).
>>>>>>>
>>>>>>> The past
>>>>>>> ========
>>>>>>>
>>>>>>> The Avocado project[1] has been working, for quite some time now, on a
>>>>>>> "set of tools and libraries" with the goal of making writing tests
>>>>>>> easier.  It is supposed to be a framework agnostic to the exact
>>>>>>> software that will be under test.
>>>>>>>
>>>>>>> But, at the same time, the Avocado project cannot deny its inheritance
>>>>>>> and influences.  Those come from Autotest[2], which had "KVM Autotest"
>>>>>>> as its largest and most developed "test".  This large Autotest test
>>>>>>> (KVM Autotest) became virt-test[3] and later got integrated into
>>>>>>> Avocado and became Avocado-VT[4] which is quite relevant here,
>>>>>>> together with its QEMU test provider[5].
>>>>>>>
>>>>>>> Avocado-VT and the QEMU test provider attempt to provide coverage
>>>>>>> across platform and QEMU versions, which increases its complexity.
>>>>>>> Also, it's built on a legacy set of principles and tools that makes
>>>>>>> some developers stir away from it.
>>>>>>>
>>>>>>> What's new?
>>>>>>> ===========
>>>>>>>
>>>>>>> A few months ago, the Avocado developers returned to its
>>>>>>> "virtualization origins", in an attempt to learn from the QEMU
>>>>>>> project, and try to help with a way to have more functional tests in
>>>>>>> the upstream QEMU repo.
>>>>>>>
>>>>>>> We believe it's possible to expand the test coverage for QEMU by
>>>>>>> facilitating
>>>>>>> the creation of more functional tests QEMU.  This is no different than 
>>>>>>> how
>>>>>>> other types of tests are already included in the tree itself.
>>>>>>>
>>>>>>> How
>>>>>>> ===
>>>>>>>
>>>>>>> How we did it (so far)
>>>>>>> ----------------------
>>>>>>>
>>>>>>> We're aware that there's a dilemma here: to be able to easily write
>>>>>>> more powerful tests, a lot of the complexity has to be moved
>>>>>>> elsewhere.  Here, it means moving complexity from the test itself to a
>>>>>>> framework.  The QEMU source tree itself has proofs of this approach,
>>>>>>> being the "scripts" and "tests/qemu-iotests" some of the examples.
>>>>>>>
>>>>>>> Avocado itself[1] provides a lot of the code that should help to
>>>>>>> absorb some of the complexities in writing tests, but not exactly
>>>>>>> everything that is needed for QEMU.  The approach we believe will have
>>>>>>> the best balance is to reuse upstream Avocado libraries whenever they
>>>>>>> are useful and generic enough, and on top of that, libraries that are
>>>>>>> part of QEMU itself.
>>>>>>>
>>>>>>> How can you get started with it
>>>>>>> -------------------------------
>>>>>>>
>>>>>>> First of all, get Avocado installed.  Besides the Avocado test runner
>>>>>>> itself, this will give you the basic libraries on which the other part
>>>>>>> of this work was built on.  We want that to be simple and painless, so
>>>>>>> here's our best bet for a one-liner installation:
>>>>>>>
>>>>>>>   pip install --user avocado-framework
>>>>>>> avocado-framework-plugin-varianter-yaml-to-mux aexpect
>>>>>>>
>>>>>>> That will install Avocado within the user's home directory.  If you
>>>>>>> give up on it, it can be uninstalled with another simple one-liner:
>>>>>>>
>>>>>>>   pip uninstall -y avocado-framework
>>>>>>> avocado-framework-plugin-varianter-yaml-to-mux aexpect
>>>>>>>
>>>>>>> Now, suppose you're working on a given feature, and want to try your
>>>>>>> luck writing a test using this work.  To avoid having you fetching and
>>>>>>> rebasing from our currently in development fork[6] and branch[7], you
>>>>>>> can just
>>>>>>> add one commit to your tree with:
>>>>>>>
>>>>>>>   curl
>>>>>>> https://patch-diff.githubusercontent.com/raw/apahim/qemu/pull/17.patch |
>>>>>>> git am -
>>>>>>>
>>>>>>> This will get a simple patch from a snapshot branch[8].  You can, of 
>>>>>>> course,
>>>>>>> do it "the git way", fetching from that repo[6] and using the
>>>>>>> non-snapshotted branch.
>>>>>>>
>>>>>>> After that, we'd love for you to take a look at some of the existing
>>>>>>> tests[9][10] and then attempt to create test for your own use case.
>>>>>>> The basic README[11] file, and the Avocado documentation[12] are also
>>>>>>> important resources not to be missed.
>>>>>>>
>>>>>>> What's next?
>>>>>>> ============
>>>>>>>
>>>>>>> Initially, feedback is what we're looking for.  It would be greatly
>>>>>>> appreciated to understand if/how this suits (or not) use cases out
>>>>>>> there.
>>>>>>>
>>>>>>> After feedback, further refinements, and more tests are written, the
>>>>>>> Avocado developers will follow up with an initial patch series for
>>>>>>> upstream QEMU.  In such a proposal, we intend to have further
>>>>>>> integration.  Ideally in way that "configure" can be given a
>>>>>>> "--with-functional-[avocado-]tests" parameter of sorts, and a "make
>>>>>>>  [functional-]check" would seamlessly include them.
>>>>>>
>>>>>> I have a few thoughts.
>>>>>>
>>>>>> We use pytest/pexpect internally to kick off QEMU runs and monitor the
>>>>>> output (no interaction with the QEMU source tree) and I think it is
>>>>>> extremely useful. So I am all for using Python to test things and this
>>>>>> looks really well done!
>>>>>>
>>>>>
>>>>> Thanks for checking it out, and for the positive words.  Now, sorry if
>>>>> I'm missing some obvious information, but is this work of yours with
>>>>> pytest/pexpect publicly available?  I'd like to also take a look at
>>>>> that, because it does look similar to the Avocado + aexpect approach
>>>>> taken here.
>>>>
>>>> Unfortunately it's not and it would take months for us to be able to
>>>> make it available.
>>>>
>>>
>>> I see.
>>>
>>>>>
>>>>>> What I don't understand though is what this gives us compared to the
>>>>>> existing QEMU test infrastructure? Besides being able to use Python
>>>>>> and a better interface what are the main benefits? I think that is
>>>>>> something worth documenting somewhere.
>>>>>>
>>>>>
>>>>> We currently intend to *add* to the QEMU test infrastructure, not
>>>>> replace it.
>>>>
>>>> Is there a benefit of integrating it into the tree then? It's always
>>>> possible to have an out of tree testing framework.
>>>>
>>>
>>> Upstream first is just the general modus operandi that we have.  Then,
>>> we want to have as much testing as possible as early as possible.
>>>
>>> Finally, by having tests in-tree, they can be seen in different light.
>>> What I mean is that when tests are in-tree, they will (or should) really
>>> map to the current state of the code they test (per commit).  If a
>>> feature changes behavior upstream, the respective test will also need a
>>> change at the same time, and as such, will mean an *intended* change and
>>> not a regression.  With out-of-tree tests, it's pretty hard to keep this
>>> synchronization and regressions will slip in, *at least* for some time.
>>>
>>> I guess the question here is actually the opposite: do you see any
>>> problems with in-tree tests?  Since you keep out of tree tests, I would
>>> honestly like to hear about your experience.
>>
>> For us the main reason is that we want to avoid editing the fork as
>> much as possible, so we aren't going to add tests to our fork.
>>
>> The other main advantage I see is that it allows the tests to be
>> updated more easily as the changes don't need to go through the more
>> rigorous QEMU review process.
>>
>> You are correct though that synchronisation is an issue.
> 
> Actually another thing we can do is run our tests on a known working
> QEMU version with new/different guest images to test other teams work.
> For example we can find Linux bugs by running the latest internal
> Linux kernel builds on our known working master QEMU.
> 
> That might be harder to setup and justify with an in tree framework,
> but isn't really a justification to make it out of tree.
> 

Sure, but it means your tests are written the way they work with different qemu 
versions. For those we have `avocado-vt` with lots of tests usually written by 
QA guys (as Avocado-vt is not easy to start with and does not attract many qemu 
developers). The beauty of Avocado_qemu is it's suppose to be very simple to 
start with to allow writing simple, specific, usually single-purpose tests, 
especially regression, tests that can be easily attached along with the fix. 
(at least that's what I see as the main goal)

Regards,
Lukáš

>>
>> Alistair
>>
>>>
>>>>>
>>>>> The benefits we envision are, besides hopefully easier and more capable
>>>>> interfaces, to simply have more upstream tests.  This means avoiding new
>>>>> regressions and improving coverage.
>>>>>
>>>>>> Also, it looks like this will require images checked into git
>>>>>> somewhere is that correct? Is there a good plan on how to handle that?
>>>>>>
>>>>>
>>>>> It won't require images checked into git.  Right now, tests use the
>>>>> vmimage library:
>>>>>
>>>>> http://avocado-framework.readthedocs.io/en/57.0/api/utils/avocado.utils.html#avocado.utils.vmimage.get
>>>>>
>>>>> Which downloads (and caches) images from external sources.
>>>>
>>>> Ah! That's cool. Managing images is one of the challenges we have at the 
>>>> moment.
>>>>
>>>> Alistair
>>>>
>>>
>>> Nice that you like it.  We also find a library such as "vmimage" is
>>> something simple enough, but still not quite something that would live
>>> in the QEMU tree.
>>>
>>> - Cleber.
>>>
>>>>>
>>>>> Please let me know if you have more questions!
>>>>> - Cleber.
>>>>>
>>>>>> Alistair
>>>>>>
>>>>>>>
>>>>>>> Thanks!
>>>>>>>
>>>>>>> References
>>>>>>> ==========
>>>>>>>
>>>>>>>  [1] http://avocado-framework.github.io/
>>>>>>>  [2] http://autotest.github.io/
>>>>>>>  [3] https://github.com/autotest/virt-test
>>>>>>>  [4] https://github.com/avocado-framework/avocado-vt
>>>>>>>  [5] https://github.com/autotest/tp-qemu
>>>>>>>  [6] https://github.com/apahim/qemu
>>>>>>>  [7] https://github.com/apahim/qemu/tree/avocado_qemu
>>>>>>>  [8] https://github.com/apahim/qemu/tree/avocado_qemu_snapshot
>>>>>>>  [9]
>>>>>>> https://github.com/apahim/qemu/blob/avocado_qemu/tests/avocado/test_info_memdev_host_nodes.py
>>>>>>> [10]
>>>>>>> https://github.com/apahim/qemu/blob/avocado_qemu/tests/avocado/test_ovmf_with_240_vcpus.py
>>>>>>> [11]
>>>>>>> https://github.com/apahim/qemu/blob/avocado_qemu/tests/avocado/README.rst
>>>>>>> [12] http://avocado-framework.readthedocs.io/
>>>>>>>
>>>>>>> --
>>>>>>> Cleber Rosa
>>>>>>> [ Sr Software Engineer - Virtualization Team - Red Hat ]
>>>>>>> [ Avocado Test Framework - avocado-framework.github.io ]
>>>>>>> [  7ABB 96EB 8B46 B94D 5E0F  E9BB 657E 8D33 A5F2 09F3  ]
>>>>>>>
>>>>>
>>>>> --
>>>>> Cleber Rosa
>>>>> [ Sr Software Engineer - Virtualization Team - Red Hat ]
>>>>> [ Avocado Test Framework - avocado-framework.github.io ]
>>>>> [  7ABB 96EB 8B46 B94D 5E0F  E9BB 657E 8D33 A5F2 09F3  ]
>>>
>>> --
>>> Cleber Rosa
>>> [ Sr Software Engineer - Virtualization Team - Red Hat ]
>>> [ Avocado Test Framework - avocado-framework.github.io ]
>>> [  7ABB 96EB 8B46 B94D 5E0F  E9BB 657E 8D33 A5F2 09F3  ]

signature.asc
Description: OpenPGP digital signature