Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests

[Ademar Reis]

On Thu, Mar 08, 2012 at 03:24:15PM -0600, Anthony Liguori wrote:
> On 03/08/2012 03:02 PM, Ademar Reis wrote:
> >On Thu, Mar 08, 2012 at 01:16:58PM -0600, Anthony Liguori wrote:
> >>On 03/08/2012 11:59 AM, Ademar Reis wrote:
> >
> ><snip>
> >
> >>
> >>>>I expect QEMU to grow tests for anything that involves launching
> >>>>QEMU directly.  Where I would not see QEMU growing tests for is
> >>>>things like launching QEMU through libvirt.
> >>>
> >>>Agree. For QEMU developers, libvirt should not be on the way, the
> >>>interaction should be minimal or non-existent.
> >>>
> >>>That's an area which will require some work in libautotest,
> >>>because due to previous QE requirements, it now invokes libvirt
> >>>methods instead of QEMU directly for a lot of stuff. But it can
> >>>be done and is part of the plan.
> >>
> >>If you're talking about libautotest as an abstraction layer for
> >>launching QEMU, I don't think that's something we should use in
> >>upstream QEMU.  Abstraction layers are okay when the things you are
> >>abstracting are well defined and relatively static.  We're talking
> >>about testing the tip of a development tree though and coordinating
> >>changes with an abstraction layer would be counter productive.
> >>
> >
> >Valid point. If I'm a qemu developer and I want to make huge
> >changes, ideally I shouldn't have to submit patches to
> >libautotest because the tests in qemu.org will break.
> >
> >I don't know how likely such scenario is (Lucas is maitaining
> >kvm-autotest for years now and the integration hardly breaks)
> >But we do have a conflict of interests, so lets evaluate a
> >few options:
> 
> Because kvm-autotest doesn't have thousands of tests and isn't part
> of 200+ developers fast paths :-)
> 
> >
> >My main interests are:
> >     - remove barriers for developers to write tests
> 
> This is my number one priority.
> 
> >     - share the testing effort between multiple projects (qemu,
> >       libvirt, ovirt, spice, QE, etc...)
> 
> This is my number 99 priority.

You have the right to consider the interests of qemu only, but
qemu is used by several other teams who depend on it to implement
their own tests. Some collaboration would be extremely beneficial
for everybody.

Collaboration goes both ways: you get benefits from the
interaction and code reuse with different projects and teams, but
it may require a few tradeoffs.

> 
> I think you could achieve practical code sharing by decomposing
> kvm-autotest into a set of useful, independent tools.  For
> instance....
> 
> >Now the options:
> >
> >1. We keep the invocation stuff in libautotest and when writting
> >a complex test in qemu.git, the test writter can choose to use it
> >because of the goodies there.
> >
> >Pros:
> >   - Lots of codepaths implemented both in python and as cmd-line
> >     utilities: less lines of code to write a test, smaller
> >     barrier for the developer.
> 
> I've got an existence proof that this is not true.  The qemu-test
> tests are smaller than the corresponding autotest tests.

You're comparing developer-level tests with the existent QA-level
tests (much more complex).

We're talking about an approach where you'll need just a few
lines of code to instantiate a VM and interact with it.

> 
> >   - Mature code in the test library.
> >   - Goodies such as video-recording and system info collection
> >     in a standard way for different projects.
> 
> Why does video-recording have to be part of libautotest?  Why can't
> you just have a simply utility that connects to a VNC session and
> records it?  Presumably that's all it's doing here.
> 
> Likewise, there are dozens of "system info collection" such as
> sosreport..  why fold this all into libautotest?

For standardization: think outside of the qemu community. A lot
of our developers contribute code to multiple projects. It gets
easier if there's a standard way to achieve the same results when
writting tests for different projects.

Anyway, it's a minor issue.

> 
> >   - Code shared with other teams.
> >   - The same test code can run on old platforms because
> >     libautotest has a compatibility layer, thus enabling a
> >     larger grid of tests.
> 
> This is not a requirement from a QEMU perspective nor do I think it's useful.

Well, it's not a requirement, but you can't expect QE teams to
spend their time on tests that work only on HEAD. It makes no
sense from their point of view.


> 
> >   - QE teams will be closer to the development, sharing
> >     some of their testing code and maintenance burden.
> 
> The QE team could simply contribute tests to qemu.git to achieve this same 
> goal.

They won't help you if you don't help them. Again, it doesn't
make sense for QE to work on HEAD. At least not the QE teams I'm
talking about.

And I say from my own experience: maintaining a set of thousands
of tests is far from trivial. You'll waste a lot of potential
contributions by alienating the fully-dedicated teams currently
working on QE.

> 
> >2. We "move" (or implement) all the libautotest stuff related to
> >qemu invocation and interaction into qemu.git, forbidding the
> >dependency of autotest in qemu.git.
> 
> Let me just be really clear here.  In order to do TDD, tests have to
> be a mandatory part of the QEMU build process.  We cannot introduce
> new dependencies in order to build tests.
> 
> Making autotest an unconditional dependency of qemu.git is a non-starter for 
> me.

You have dependencies on gtest and other libraries. it's not that
different.

Plus it's not unconditional: the test runner will report tests
SKIPPED if a dependency is not present.

It's actually the oposite: you'll have to forbid developers from
submiting tests that use libautotest if you don't want the
dependency.

> 
> >In a way, that's what tends to happen with qemu-test if things
> >keep going on AS IS in qemu.org (I'm absolutely sure that you'll
> >replicate a lot of what autotest does as you increase the test
> >coverage and complexity).
> >
> >Pros:
> >   - Everything under qemu.git control: you break it, you fix it.
> >   - Organic growth of the test infra-structure in QEMU
> >
> >Cons:
> >   - Lot of code will be duplicated to cover the main code paths:
> >     writting tests will require writting/supporting considerable
> >     ammount of code (that already exists in autotest).
> 
> Again, existence proof that this isn't true.

Well, let's agree to disagree then. My bet is that qemu-test will
suffer a lot of growing pains if TDD is properly used, but I can't
prove it.

> 
> >   - QE will be alienated from the qemu test effort. There will be
> >     no integration between the QE efforts and the maintenance of
> >     the qemu developer-level tests.
> 
> I think we're a pretty friendly and open community :-)  There is no
> reason that QE should be "alienated" unless folks are choosing not
> to participate upstream.

For the exact same reasons you as a developer don't want to
implement tests inside autotest, QE won't want to implement tests
for qemu.git. It's out of their comfort zone, just put yourself
on their shoes.

> 
> >   - You don't get the goodies from autotest (standardized system
> >     info collection, video recording, grid support, etc).
> >   - The new tests will work only on the master branch.
> 
> This last one is a legitimate point that I have considered myself.
> But I think the value of having tests in HEAD outweigh the cost
> here.

Thinking about qemu only, agree.

> 
> >3. A mix of (1) and (2): we "move" everything under qemu.git, but
> >keep the compatibility layer and provide a "libqemutest" for
> >third-parties.
> >
> >Anybody writting a test that interacts with qemu will use this
> >library: qemu, libvirt, spice, ovirt, QE tests, etc.
> 
> I really see this all as over architecting to be honest.
> 
> Can someone explain in clear terms (without appealing to maturity,
> flexibility, or any other qualitative aspect) why it takes anything
> more than a few dozen shell functions to write all of the tests that
> are in kvm-autotest test today?

This is close to being offensive IMO, so I won't answer. :-)

> 
> You launch qemu (by just using a command), and you interact with the
> monitor via QMP and HMP.  Why do you ever need anything more than
> that?

Well, again, I'll agree to disagree.

> 
> Why does libautotest even need to exist?

You're looking at things from the sole point of view of a qemu
developer. If you don't want to see things from a different
perspective and accept any kind of tradeoffs for better
collaboration with other teams, then unfortunately there's not
much we can do. It's a pity, because collaboration goes both
ways.

We're trying very hard to satisfy your requirements and
interests. Don't you see any way you can help us satisfy
ours?

-- 
Ademar de Souza Reis Jr.
Red Hat

^[:wq!