Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests

[Ademar Reis]

On Thu, Mar 08, 2012 at 09:14:02AM -0600, Anthony Liguori wrote:
> On 03/08/2012 09:07 AM, Ademar Reis wrote:
> >On Thu, Mar 08, 2012 at 08:56:23AM -0600, Anthony Liguori wrote:
> >>On 03/08/2012 08:49 AM, Ademar Reis wrote:
> >>>On Thu, Mar 08, 2012 at 07:36:11AM -0600, Anthony Liguori wrote:
> >>>>On 03/07/2012 10:00 PM, Lucas Meneghel Rodrigues wrote:
> >>>>>Virt/qemu tests: Minimal guest images
> >>>>>-------------------------------------
> >>>>>
> >>>>>In order to make development level test possible, we need the tests to 
> >>>>>run fast.
> >>>>>In order to do that, a set of minimal guest images is being developed 
> >>>>>and we
> >>>>>have a version for x86_64 ready and functional:
> >>>>>
> >>>>>https://github.com/autotest/buildroot-autotest
> >>>>
> >>>>I'm really not a fan of buildroot.  Note that in order to ship
> >>>>binaries, full source needs to be provided in order to comply with
> >>>>the GPL.  The FSF at least states that referring to another website
> >>>>for source that's not under your control doesn't satisfy the
> >>>>requirements of the GPL.
> >>>>
> >>>>Just out of curiosity, did you try to use qemu-test?  Is there a
> >>>>reason you created something different?
> >>>>
> >>>>I think it's good that you're thinking about how to make writing
> >>>>tests easier, but we have a growing test infrastructure in QEMU and
> >>>>that's what I'd prefer people focused on.
> >>>>
> >>>
> >>>You probably remember the long thread we had back in December on
> >>>qemu-devel on this topic. Back then our message was "we have a
> >>>growing test infrastructure in s/QEMU/autotest/ and that's what
> >>>we'd prefer people focused on". :-)
> >>>
> >>> From Dor:
> >>>
> >>>(http://lists.nongnu.org/archive/html/qemu-devel/2011-12/msg03024.html)
> >>>
> >>>"""
> >>>If you wish, you can challenge Lucas and Cleber w/ these type of
> >>>requirements and we'll all improve as a result.
> >>>"""
> >>>
> >>>Your response was:
> >>>
> >>>"""
> >>>Well consider qemu-test the challenge. It's an existence proof
> >>>that we can have a very easy to use framework for testing that
> >>>runs extremely fast and is very easy to write tests for.
> >>>"""
> >>>
> >>>http://knowyourmeme.com/memes/challenge-accepted ;-)
> >>>
> >>>I particularly agreed with basically everything you said on that
> >>>discussion regarding test simplification (I had just joined the
> >>>team back then). To me, autotest has been focusing on QE-level,
> >>>leaving the developer-level test requirements out. Now we're
> >>>attacking this new front, and a lot of the requirements are
> >>>indeed from that discussion.
> >>
> >>If you want to talk about this in terms of "requirements", my
> >>requirement is for "developer-level" tests to live in qemu.git and
> >>be integrated into make check.
> >>
> >>Just as we've been discussing and working on since the previous set of 
> >>discussions.
> >>
> >>>By simplifying the design and bringing barriers down, we hope to
> >>>reach a broader audience and help developers write and maintain
> >>>tests, benefiting from all the instrumentation that autotest
> >>>brings. It's not going to be just about qemu (check the new test
> >>>examples).
> >>>
> >>>We have a team fully dedicated to autotest and it's used not only
> >>>by Qemu but also libvirt, Google, Xen, Fedora, Twitter, etc, etc
> >>>(these all have code contributions in autotest)
> >>>
> >>>That said, the current qemu-tests will probably be easily
> >>>integrated into (the new) autotest and we hope that, given enough
> >>>time, autotest will be good enough to relieve qemu from the
> >>>framework maintenance and code duplication with other projects.
> >>
> >>autotest should not be the focal point for integration.  qemu.git should be.
> >>
> >>I'd be perfectly happy to review patches submitting the test
> >>infrastructure from kvm-autotest into qemu.git (provided it didn't
> >>have unreasonable external dependencies and fit into QEMU).
> >>
> >>Developer-level tests need to live where the developers live.  The
> >>developers live in qemu.git.  See my other response on this thread
> >>for the explanation of why this is so important.
> >>
> >
> >Excelent, we're in the same page then. This was my number 1
> >requirement when I was discussing the changes with Lucas and
> >Cleber. For convenience, I'll repeat here what I wrote in a
> >previous e-mail (no qemu-devel archive available yet to use as a
> >reference).
> >
> >In summary, autotest is (or is going to be) a framework that
> >provides:
> >
> >  - A test runner, with grid/cluster support and advanced
> >    instrumentation
> >  - A devel library and set of utilities for test writers
> >  - A set of pre-built images (JeOS – Just Enough OS) for
> >    test writers
> 
> I don't think autotest is the right place for this to live.  We need
> this directly in qemu.git otherwise we're severely limited in what
> tests we can write.
> 
> I guess that doesn't preclude autotest having its own JeOS
> mechanism, but we clearly need one in qemu.git.
> 
> >
> >(attached is a picture showing what we want to achieve)
> >
> >If a project has an internal library or set of utilities that can
> >be of general use, they can be submitted to autotest.git for
> >inclusion, thus reaching a broader audience.
> >
> >A short summary of the plans:
> >
> >  - Tests can live anywhere and each devel team implements and
> >    maintains their own set of tests
> 
> Let me change this to:
> 
> - Autotest will learn how to harness the tests that each development
> team creates in their respective git repository.

Yep, given this simple requirement: "tests return 0 or an error
code". The output from the test runner will be a simple PASS/FAIL
and stdout/stderr will be properly collected by the test runner
and made available in a standard location. If using TAP, even
better.

This will allow trivial tests. As the test complexity grows, then
developers may start using parts of the autotest library or
utilities, thus requiring it installed (a trivial bootstrap
procedure is also a requirement, see the e-mail from Lucas).

> 
> I think this is what you mean, but it's not how I interpret the
> above.  I read this as, "tests can (and should) live anywhere across
> any set of repositories".  I think when you say "live anywhere", you
> mean this strictly from an autotest perspective and are trying to
> accomodate tests that live in upstream repositories.

> >  - Usage of the autotest library by test writers is optional
> >  - Tests are scripts returning 0 or error (any language)

I hope these two clarify the above.

> >  - Tests can be run individually or in sets
> >  - Tests should run fast, our target is seconds or a few minutes
> >  - The test runner is smart and “just works” by default
> >    - Trivial standard output (FAIL, PASS, SKIPPED)
> >    - Collect logs, OS data and other stuff (e.g. --record-video!)
> >    - Skip unsupported tests based on the environment they're run
> >    - Multiplex configurations / platforms when on the grid
> >    - Support to run tests “in the cloud”
> 
> This is where you start to lose me.  If all you're saying is, "QEMU
> can continue to use gtest to build out it's test infrastructure and
> autotest will learn how to use it", then we're in violent agreement.

Good :-)

Inside qemu.git, you can add as many instrumentation and helper
libraries as you want. As a developer it'll be up to you to setup
your environment to run the tests, and autotest will be just yet
another dependency.

QE will probably be interested in setting up a bot with several
tests from differente repositories, but that's their problem, not
yours.

BTW, please note that we're not trying to cover unit-tests in
autotest. At least not by the definition of unit-tests that I
know, which are code-level tests without running the application
itself (unit-tests test functions/methods only, not the
application as a whole).

Autotest is more about integration tests, which are complementary
to unit-tests in a "test driven development" environment: you
instantiate your program by using scripts and check if things
happen as you expect, as in this trivial example:

qemu-img-convert.sh
    #!/bin/bash
    qemu-img convert -O qcow2 $DATA/qemu-imgs/reference.vdi 
$TEMPDIR/output.qcow2
    exec diff -q $TEMPDIR/output.qcow2 $DATA/qemu-imgs/reference.qcow2

(of course, you could encapsulate your unit-test runner as one
test inside autotest)

The way I see test automation working at the developer level is
similar to what we have in WebKit (my previous project): dozens
of thousands of small test scripts (28k last time I counted), run
all the time on a grid of buildbots and on developer machines.
Tests are maintained by the devel team and constantly growing
because bugfixes and new features are acompanied by new tests.

And a broken test is considered as bad as a broken build.

BUT we'll need a lot of effort and time to get there. We hope
autotest will help somehow, but it's just a small baby step.

Cheers,
   - Ademar

-- 
Ademar de Souza Reis Jr.
Red Hat

^[:wq!