Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests

[Anthony Liguori]

On 03/08/2012 10:05 AM, Ademar Reis wrote:
> On Thu, Mar 08, 2012 at 09:14:02AM -0600, Anthony Liguori wrote:
> > On 03/08/2012 09:07 AM, Ademar Reis wrote:
> > > On Thu, Mar 08, 2012 at 08:56:23AM -0600, Anthony Liguori wrote:
> > > > On 03/08/2012 08:49 AM, Ademar Reis wrote:
> > > > > On Thu, Mar 08, 2012 at 07:36:11AM -0600, Anthony Liguori wrote:
> > > > > > On 03/07/2012 10:00 PM, Lucas Meneghel Rodrigues wrote:
> > > > > > > Virt/qemu tests: Minimal guest images
> > > > > > > -------------------------------------
> > > > > > >
> > > > > > > In order to make development level test possible, we need the tests to run fast.
> > > > > > > In order to do that, a set of minimal guest images is being developed and we
> > > > > > > have a version for x86_64 ready and functional:
> > > > > > >
> > > > > > > https://github.com/autotest/buildroot-autotest
> > > > > >
> > > > > > I'm really not a fan of buildroot.  Note that in order to ship
> > > > > > binaries, full source needs to be provided in order to comply with
> > > > > > the GPL.  The FSF at least states that referring to another website
> > > > > > for source that's not under your control doesn't satisfy the
> > > > > > requirements of the GPL.
> > > > > >
> > > > > > Just out of curiosity, did you try to use qemu-test?  Is there a
> > > > > > reason you created something different?
> > > > > >
> > > > > > I think it's good that you're thinking about how to make writing
> > > > > > tests easier, but we have a growing test infrastructure in QEMU and
> > > > > > that's what I'd prefer people focused on.
> > > > >
> > > > > You probably remember the long thread we had back in December on
> > > > > qemu-devel on this topic. Back then our message was "we have a
> > > > > growing test infrastructure in s/QEMU/autotest/ and that's what
> > > > > we'd prefer people focused on". :-)
> > > > >
> > > > >  From Dor:
> > > > >
> > > > > (http://lists.nongnu.org/archive/html/qemu-devel/2011-12/msg03024.html)
> > > > >
> > > > > """
> > > > > If you wish, you can challenge Lucas and Cleber w/ these type of
> > > > > requirements and we'll all improve as a result.
> > > > > """
> > > > >
> > > > > Your response was:
> > > > >
> > > > > """
> > > > > Well consider qemu-test the challenge. It's an existence proof
> > > > > that we can have a very easy to use framework for testing that
> > > > > runs extremely fast and is very easy to write tests for.
> > > > > """
> > > > >
> > > > > http://knowyourmeme.com/memes/challenge-accepted ;-)
> > > > >
> > > > > I particularly agreed with basically everything you said on that
> > > > > discussion regarding test simplification (I had just joined the
> > > > > team back then). To me, autotest has been focusing on QE-level,
> > > > > leaving the developer-level test requirements out. Now we're
> > > > > attacking this new front, and a lot of the requirements are
> > > > > indeed from that discussion.
> > > >
> > > > If you want to talk about this in terms of "requirements", my
> > > > requirement is for "developer-level" tests to live in qemu.git and
> > > > be integrated into make check.
> > > >
> > > > Just as we've been discussing and working on since the previous set of 
> > > > discussions.
> > > >
> > > > > By simplifying the design and bringing barriers down, we hope to
> > > > > reach a broader audience and help developers write and maintain
> > > > > tests, benefiting from all the instrumentation that autotest
> > > > > brings. It's not going to be just about qemu (check the new test
> > > > > examples).
> > > > >
> > > > > We have a team fully dedicated to autotest and it's used not only
> > > > > by Qemu but also libvirt, Google, Xen, Fedora, Twitter, etc, etc
> > > > > (these all have code contributions in autotest)
> > > > >
> > > > > That said, the current qemu-tests will probably be easily
> > > > > integrated into (the new) autotest and we hope that, given enough
> > > > > time, autotest will be good enough to relieve qemu from the
> > > > > framework maintenance and code duplication with other projects.
> > > >
> > > > autotest should not be the focal point for integration.  qemu.git should be.
> > > >
> > > > I'd be perfectly happy to review patches submitting the test
> > > > infrastructure from kvm-autotest into qemu.git (provided it didn't
> > > > have unreasonable external dependencies and fit into QEMU).
> > > >
> > > > Developer-level tests need to live where the developers live.  The
> > > > developers live in qemu.git.  See my other response on this thread
> > > > for the explanation of why this is so important.
> > >
> > > Excelent, we're in the same page then. This was my number 1
> > > requirement when I was discussing the changes with Lucas and
> > > Cleber. For convenience, I'll repeat here what I wrote in a
> > > previous e-mail (no qemu-devel archive available yet to use as a
> > > reference).
> > >
> > > In summary, autotest is (or is going to be) a framework that
> > > provides:
> > >
> > >   - A test runner, with grid/cluster support and advanced
> > >     instrumentation
> > >   - A devel library and set of utilities for test writers
> > >   - A set of pre-built images (JeOS – Just Enough OS) for
> > >     test writers
> >
> > I don't think autotest is the right place for this to live.  We need
> > this directly in qemu.git otherwise we're severely limited in what
> > tests we can write.
> >
> > I guess that doesn't preclude autotest having its own JeOS
> > mechanism, but we clearly need one in qemu.git.
> >
> > > (attached is a picture showing what we want to achieve)
> > >
> > > If a project has an internal library or set of utilities that can
> > > be of general use, they can be submitted to autotest.git for
> > > inclusion, thus reaching a broader audience.
> > >
> > > A short summary of the plans:
> > >
> > >   - Tests can live anywhere and each devel team implements and
> > >     maintains their own set of tests
> >
> > Let me change this to:
> >
> > - Autotest will learn how to harness the tests that each development
> > team creates in their respective git repository.
>
> Yep, given this simple requirement: "tests return 0 or an error
> code". The output from the test runner will be a simple PASS/FAIL
> and stdout/stderr will be properly collected by the test runner
> and made available in a standard location. If using TAP, even
> better.

We use gtest, so this will not be the case.  I think it would be best for 
autotest to learn how to interact with the gtest protocol.

> This will allow trivial tests. As the test complexity grows, then
> developers may start using parts of the autotest library or
> utilities, thus requiring it installed (a trivial bootstrap
> procedure is also a requirement, see the e-mail from Lucas).

What is the value of the autotest library to a test writer other than being part 
of the "autotest framework"?

> > This is where you start to lose me.  If all you're saying is, "QEMU
> > can continue to use gtest to build out it's test infrastructure and
> > autotest will learn how to use it", then we're in violent agreement.
>
> Good :-)
>
> Inside qemu.git, you can add as many instrumentation and helper
> libraries as you want. As a developer it'll be up to you to setup
> your environment to run the tests, and autotest will be just yet
> another dependency.

I think there's an implicit assumption here that the tests in qemu.git will have 
a dependency on autotest.  I'd like to understand why this dependency is necessary.

Normally, autotest executes third party tests, what makes QEMU special here?

> QE will probably be interested in setting up a bot with several
> tests from differente repositories, but that's their problem, not
> yours.

Right, and this is what autotest is very good at :-)

> BTW, please note that we're not trying to cover unit-tests in
> autotest. At least not by the definition of unit-tests that I
> know, which are code-level tests without running the application
> itself (unit-tests test functions/methods only, not the
> application as a whole).

There is never a clear line between unit-tests and integration tests as long as 
you're talking about a single project.

I expect QEMU to grow tests for anything that involves launching QEMU directly. 
 Where I would not see QEMU growing tests for is things like launching QEMU 
through libvirt.

> Autotest is more about integration tests, which are complementary
> to unit-tests in a "test driven development" environment: you
> instantiate your program by using scripts and check if things
> happen as you expect, as in this trivial example:
>
> qemu-img-convert.sh
>      #!/bin/bash
>      qemu-img convert -O qcow2 $DATA/qemu-imgs/reference.vdi 
> $TEMPDIR/output.qcow2
>      exec diff -q $TEMPDIR/output.qcow2 $DATA/qemu-imgs/reference.qcow2
>
> (of course, you could encapsulate your unit-test runner as one
> test inside autotest)
>
> The way I see test automation working at the developer level is
> similar to what we have in WebKit (my previous project): dozens
> of thousands of small test scripts (28k last time I counted), run
> all the time on a grid of buildbots and on developer machines.
> Tests are maintained by the devel team and constantly growing
> because bugfixes and new features are acompanied by new tests.
>
> And a broken test is considered as bad as a broken build.
>
> BUT we'll need a lot of effort and time to get there. We hope
> autotest will help somehow, but it's just a small baby step.

Yes, I have a similar view of the long term goal.  And just like those thousands 
of tests live in the main webkit tree, I think we need thousands of tests in 
qemu.git.

Regards,

Anthony Liguori

> Cheers,
>     - Ademar