Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests

[Ademar Reis]

On Thu, Mar 08, 2012 at 11:54:31AM +0000, Stefan Hajnoczi wrote:
> On Thu, Mar 8, 2012 at 11:44 AM, Stefan Hajnoczi <address@hidden> wrote:
> > On Thu, Mar 8, 2012 at 4:00 AM, Lucas Meneghel Rodrigues <address@hidden> 
> > wrote:
> >> One of our main goals is to provide useful tools for the qemu community,
> >> since we have a good number of tests and libraries written to perform
> >> integration/QA testing for that tool, being successfuly used by a number of
> >> QA teams that work on qemu. Also, we recently provided a subset of that
> >> infrastructure to test libvirt, one of our virtualization projects of
> >> interest.
> >
> > Thanks for sharing.
> 
> One thing I should have added is that my message is about what it
> would take for me to use autotest and contribute tests.  But now I
> realize that you might be going for a different model:
> 
> If you're aiming for a different model where autotest integrates
> external test suites (i.e. tests wouldn't be written in autotest.git,
> instead autotest.git would contain snapshots of external test suites),
> then this proposal seems fine.  Upstream projects like QEMU would
> develop their own test suite and it would be dropped into autotest or
> a specific autotest instance.
> 

Yes, that's the idea. We want autotest to be the framework, not
(just a) collection of tests. We also want each development team
to implement and maintain their own set of tests, using (or not)
the goodies from autotest at their discretion.

In summary, autotest is (or is going to be) a framework that
provides:

 - A test runner, with grid/cluster support and advanced
   instrumentation
 - A devel library and set of utilities for test writers
 - A set of pre-built images (JeOS – Just Enough OS) for
   test writers 

(attached is a picture showing what we want to achieve)

If a project has an internal library or set of utilities that can
be of general use, they can be submitted to autotest.git for
inclusion, thus reaching a broader audience.

A short summary of the plans:

 - Tests can live anywhere and each devel team implements and
   maintains their own set of tests
 - Usage of the autotest library by test writers is optional
 - Tests are scripts returning 0 or error (any language)
 - Tests can be run individually or in sets
 - Tests should run fast, our target is seconds or a few minutes
 - The test runner is smart and “just works” by default
   - Trivial standard output (FAIL, PASS, SKIPPED)
   - Collect logs, OS data and other stuff (e.g. --record-video!)
   - Skip unsupported tests based on the environment they're run
   - Multiplex configurations / platforms when on the grid
   - Support to run tests “in the cloud”

There are also lots of small changes and usability improvements
in the pipeline (and feedback right now is very valuable).

Thanks,
  - Ademar

-- 
Ademar de Souza Reis Jr.
Red Hat

^[:wq!