[Top][All Lists]
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization test
From: |
Anthony Liguori |
Subject: |
Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests |
Date: |
Thu, 08 Mar 2012 15:24:15 -0600 |
User-agent: |
Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.23) Gecko/20110922 Lightning/1.0b2 Thunderbird/3.1.15 |
On 03/08/2012 03:02 PM, Ademar Reis wrote:
On Thu, Mar 08, 2012 at 01:16:58PM -0600, Anthony Liguori wrote:
On 03/08/2012 11:59 AM, Ademar Reis wrote:
<snip>
I expect QEMU to grow tests for anything that involves launching
QEMU directly. Where I would not see QEMU growing tests for is
things like launching QEMU through libvirt.
Agree. For QEMU developers, libvirt should not be on the way, the
interaction should be minimal or non-existent.
That's an area which will require some work in libautotest,
because due to previous QE requirements, it now invokes libvirt
methods instead of QEMU directly for a lot of stuff. But it can
be done and is part of the plan.
If you're talking about libautotest as an abstraction layer for
launching QEMU, I don't think that's something we should use in
upstream QEMU. Abstraction layers are okay when the things you are
abstracting are well defined and relatively static. We're talking
about testing the tip of a development tree though and coordinating
changes with an abstraction layer would be counter productive.
Valid point. If I'm a qemu developer and I want to make huge
changes, ideally I shouldn't have to submit patches to
libautotest because the tests in qemu.org will break.
I don't know how likely such scenario is (Lucas is maitaining
kvm-autotest for years now and the integration hardly breaks)
But we do have a conflict of interests, so lets evaluate a
few options:
Because kvm-autotest doesn't have thousands of tests and isn't part of 200+
developers fast paths :-)
My main interests are:
- remove barriers for developers to write tests
This is my number one priority.
- share the testing effort between multiple projects (qemu,
libvirt, ovirt, spice, QE, etc...)
This is my number 99 priority.
I think you could achieve practical code sharing by decomposing kvm-autotest
into a set of useful, independent tools. For instance....
Now the options:
1. We keep the invocation stuff in libautotest and when writting
a complex test in qemu.git, the test writter can choose to use it
because of the goodies there.
Pros:
- Lots of codepaths implemented both in python and as cmd-line
utilities: less lines of code to write a test, smaller
barrier for the developer.
I've got an existence proof that this is not true. The qemu-test tests are
smaller than the corresponding autotest tests.
- Mature code in the test library.
- Goodies such as video-recording and system info collection
in a standard way for different projects.
Why does video-recording have to be part of libautotest? Why can't you just
have a simply utility that connects to a VNC session and records it? Presumably
that's all it's doing here.
Likewise, there are dozens of "system info collection" such as sosreport.. why
fold this all into libautotest?
- Code shared with other teams.
- The same test code can run on old platforms because
libautotest has a compatibility layer, thus enabling a
larger grid of tests.
This is not a requirement from a QEMU perspective nor do I think it's useful.
- QE teams will be closer to the development, sharing
some of their testing code and maintenance burden.
The QE team could simply contribute tests to qemu.git to achieve this same goal.
2. We "move" (or implement) all the libautotest stuff related to
qemu invocation and interaction into qemu.git, forbidding the
dependency of autotest in qemu.git.
Let me just be really clear here. In order to do TDD, tests have to be a
mandatory part of the QEMU build process. We cannot introduce new dependencies
in order to build tests.
Making autotest an unconditional dependency of qemu.git is a non-starter for me.
In a way, that's what tends to happen with qemu-test if things
keep going on AS IS in qemu.org (I'm absolutely sure that you'll
replicate a lot of what autotest does as you increase the test
coverage and complexity).
Pros:
- Everything under qemu.git control: you break it, you fix it.
- Organic growth of the test infra-structure in QEMU
Cons:
- Lot of code will be duplicated to cover the main code paths:
writting tests will require writting/supporting considerable
ammount of code (that already exists in autotest).
Again, existence proof that this isn't true.
- QE will be alienated from the qemu test effort. There will be
no integration between the QE efforts and the maintenance of
the qemu developer-level tests.
I think we're a pretty friendly and open community :-) There is no reason that
QE should be "alienated" unless folks are choosing not to participate upstream.
- You don't get the goodies from autotest (standardized system
info collection, video recording, grid support, etc).
- The new tests will work only on the master branch.
This last one is a legitimate point that I have considered myself. But I think
the value of having tests in HEAD outweigh the cost here.
3. A mix of (1) and (2): we "move" everything under qemu.git, but
keep the compatibility layer and provide a "libqemutest" for
third-parties.
Anybody writting a test that interacts with qemu will use this
library: qemu, libvirt, spice, ovirt, QE tests, etc.
I really see this all as over architecting to be honest.
Can someone explain in clear terms (without appealing to maturity, flexibility,
or any other qualitative aspect) why it takes anything more than a few dozen
shell functions to write all of the tests that are in kvm-autotest test today?
You launch qemu (by just using a command), and you interact with the monitor via
QMP and HMP. Why do you ever need anything more than that?
Why does libautotest even need to exist?
Regards,
Anthony Liguori
- Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests, (continued)
- Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests, Ademar Reis, 2012/03/08
- Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests, Ademar Reis, 2012/03/08
- Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests, Anthony Liguori, 2012/03/08
- Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests, Ademar Reis, 2012/03/08
- Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests, Anthony Liguori, 2012/03/08
- Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests, Ademar Reis, 2012/03/08
- Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests, Lucas Meneghel Rodrigues, 2012/03/08
- Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests, Lucas Meneghel Rodrigues, 2012/03/08
- Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests, Anthony Liguori, 2012/03/08
- Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests, Ademar Reis, 2012/03/08
- Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests,
Anthony Liguori <=
- Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests, Ademar Reis, 2012/03/08
- Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests, Anthony Liguori, 2012/03/08
- Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests, Ademar Reis, 2012/03/08
- Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests, Stefan Hajnoczi, 2012/03/09
- Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests, Ademar Reis, 2012/03/09
- Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests, Stefan Hajnoczi, 2012/03/09
- Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests, Ademar Reis, 2012/03/09
- Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests, Stefan Hajnoczi, 2012/03/09
- Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests, Kevin Wolf, 2012/03/09
- Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests, Anthony Liguori, 2012/03/09