Re: [Qemu-devel] [ANNOUNCE] qemu-test: a set of tests scripts for QEMU

[Avi Kivity]

On 12/28/2011 04:27 PM, Anthony Liguori wrote:
>> Maybe I've used the wrong wording. I got the feeling that, besides
>> testing qemu
>> the way you need it, keeping qemu-test simple was really important.
>
>
> Simple is always important.  In the case of qemu-test, there are some
> important trade offs made to keep things simple and fast.  It doesn't
> try to work with arbitrary guests which means the tests need to handle
> only one environment.  The guest is pre-made to have exactly what is
> needed for the tests so there is no setup required.  The guest is as
> small as possible such that the test can run as quickly as possible.

In fact using linux as a guest negates that.  First of all, which linux
version? if it's fixed, you'll eventually miss functionality and need to
migrate.  If it keeps changing, so does your test, and it will keep
breaking.

Using Linux as a guest allows you to do system tests (ping was an
example) but doesn't allow you to do unit tests (test regression where
if this bit in that register was set, but if them bits in thar registers
were clear, then the packet would be encoded with ROT-26 before being
sent out).

I think qemu-test needs to use its own drivers which allow full control
over what you do with the hardware.  Otherwise, a regression that only
shows up in non-Linux guests will not be testable with qemu-test.

> There is no need to have a single tool that meets every possible
> need.  In fact, the Unix tradition is to have separate single purposed
> tools.

Those single purpose tools, if successful, tend to grow more purposes
(cf. qemu), and if unsuccessful, tend to lose purpose.

> Having two test tools it not a bad thing provided that the overlap
> isn't significant.  

This is important, if the boundary isn't clear, then it will grow more
fuzzy in time.

I suggest the following:

 - qemu-test: qemu unit tests
 - kvm-unit-tests: kvm unit tests
 - kvm-autotest: unit test drivers + system tests

> We shouldn't be discussing whether it's possible to merge the two
> tools, but rather what the technical benefits of doing so would be.
>
> Since at this point, there is almost no overlap between the two, I
> don't see any actual technical benefit to merging them.  I see benefit
> to autotest executing qemu-test, of course.

I'd say that running a ping test is a weak version of kvm-autotest's
system tests.  Running a synthetic test that pokes values into memory
and mmio and sees a packet coming out is a unit test (the latter can in
fact be executed without a guest at all, just a table driving calls to
the memory and irq APIs).


> Just putting the code in kvm-autotest.git in a directory doesn't make
> sense to me.  Beyond the lack of a technical reason to do so,
> logistically, it makes it harder for me to ask people to submit test
> cases with a patch series if I can't actually apply the test case when
> I'm applying the patches to qemu.git.
>
> If qemu-test didn't use large submodules (like linux.git), I would
> have made qemu-test part of qemu.git.  As far as I'm concerned,
> qemu-test.git is just an extension to qemu.git.

Why not just put it in qemu.git?

-- 
error compiling committee.c: too many arguments to function