Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests

[Ademar Reis]

On Thu, Mar 08, 2012 at 08:48:33AM -0600, Anthony Liguori wrote:
> On 03/08/2012 08:01 AM, Lucas Meneghel Rodrigues wrote:
> >On 03/08/2012 10:36 AM, Anthony Liguori wrote:
> >
> >>>Virt/qemu tests: Minimal guest images
> >>>-------------------------------------
> >>>
> >>>In order to make development level test possible, we need the tests to
> >>>run fast.
> >>>In order to do that, a set of minimal guest images is being developed
> >>>and we
> >>>have a version for x86_64 ready and functional:
> >>>
> >>>https://github.com/autotest/buildroot-autotest
> >>
> >>I'm really not a fan of buildroot. Note that in order to ship binaries,
> >>full source needs to be provided in order to comply with the GPL. The
> >>FSF at least states that referring to another website for source that's
> >>not under your control doesn't satisfy the requirements of the GPL.
> >
> >We have a full clone of the buildroot repository that points out to the 
> >sources,
> >if it's necessary to have a clone of all the projects needed host there in 
> >order
> >to be able to publish a binary image to help people, then we can do it.
> 
> This is harder than I think you anticipate but okay..
> 
> >
> >>Just out of curiosity, did you try to use qemu-test? Is there a reason
> >>you created something different?
> >
> >I did, and it does what it proposes to. Nothing against it, but we have code
> >that can do more things, that has been developed for longer time.
> >
> >It's similar to qemu-jeos vs buildbot, you have written scripts to create an
> >image, which happens to be precisely why buildroot was written more than 10
> >years ago and it works very well, allowing me to put things on the image that
> >are not possible with qemu-jeos. If the problem is to point out to all sub
> >modules as git repos, we can make that happen too, rather than re-writing 
> >stuff
> >that works.
> >
> >For a long time I would like to see people working on a single code base,
> 
> I agree, we just disagree on what that code base should be :-)
> 
> That code base should be qemu.git.  This discussion isn't about
> improving third-party QE--at least not to me.  Third party QE is a
> solved problem thanks to all of your wonderful work with
> kvm-autotest.  I'm sure you're looking for more
> participation/developers, but even if you had twice the developers
> working on kvm-autotest, I don't think it would fundamentally change
> our quality.

You got it wrong: we don't want people to work on autotest, we
want people to use autotest because we believe there's a lot of
benefit on using it.

And we *know* that the current autotest is not good enough for
developers, which is why we're now working on new usage
scenarios.

> 
> I'm interested in driving our development process toward test driven
> development such that all 200+ people that write patches for QEMU
> for a given release write and run tests as part of their normal
> development process.  The requirements to achieve this are different
> than the requirements you have been working against up until now.

Fully agree, these are new requirements.

> 
> Every barrier that we put up to writing and running tests will
> reduce than number of 200+ to something lower.
> 
> Submitting a patch to a different project than qemu.git is a
> barrier.  Now instead of getting a single set of feedback, you've
> got to deal with feedback from two projects.
> 

Fully agree, please check my previous email with the plans for
the new architecture of autotest.

> Having to use setup another framework (that runs as root) is another
> barrier.  I change a file in QEMU, run make, then run make check.  I
> don't install anything, I don't sudo anything.  The whole process is
> relatively quick and painless.

Fully agree, this is one of the requirement we're going after.

> 
> Having to make a change to autotest, then install autotest, relaunch
> it, etc, is just too complicated to be part of a developers fast
> path.

Fully agree, which is why we're not planning any of it.

> 
> Now I think we should talk about how to make tests that live in
> qemu.git and run as part of make check easily harnessed by
> autotest..  But I think the primary focus of future test work needs
> to be within qemu.git.

Fully agree, please check our previous e-mails with the plans for
the new architecture.

So as you can see, there are no disagreements. :-) Please keep
the feedback and requirements coming.

Cheers,
  - Ademar

-- 
Ademar de Souza Reis Jr.
Red Hat

^[:wq!