Re: [Qemu-devel] [ANNOUNCE] qemu-test: a set of tests scripts for QEMU

[Avi Kivity]

On 12/28/2011 06:44 PM, Anthony Liguori wrote:
> On 12/28/2011 09:28 AM, Avi Kivity wrote:
>> On 12/28/2011 05:01 PM, Avi Kivity wrote:
>>> I'd say that running a ping test is a weak version of kvm-autotest's
>>> system tests.  Running a synthetic test that pokes values into memory
>>> and mmio and sees a packet coming out is a unit test (the latter can in
>>> fact be executed without a guest at all, just a table driving calls to
>>> the memory and irq APIs).
>>>
>>
>> Consider
>>    98d23704138e0
>>    7b4252e83f6f7d
>>    f7e80adf3cc4
>>    c16ada980f43
>>    4abf12f4ea8
>>
>> (found by looking for 'fix' in the commit log and filtering out the
>> commits that don't support my case)
>>
>> how can you reject such patches on the grounds that they're not
>> accompanied by unit tests?
>
> That's why I've also proposed qtest.  But having written quite a few
> qtest unit tests by now, you hit the limits of this type of testing
> pretty quickly.

Can you describe those limits?

>
>> only by making it easy to add tests for
>> them.  I think it would be hard/impossible to test them with
>> linux-as-a-guest, since they fix edge cases that linux doesn't invoke.
>> But by having our own driver (often just using qmp to poke at memory),
>> we can easily generate the sequence that triggers the error.
>>
>> We'd probably need a library to support setting up a pci device's BARs,
>> but that's easy with qmp/python integration.  You can even poke a small
>> executable into memory and execute it directly, if you really need guest
>> cpu interaction.
>
> Please review the qtest series.  I think it offers a pretty good
> approach to writing this style of test.  But as I mentioned, you hit
> the limits pretty quickly.

I think it's great, it looks like exactly what I wanted, except it's
been delivered on time.  I'd really like to see it integrated quickly
with some flesh around it, then replying -ENOTEST to all patches.  This
will improve qemu's quality a lot more than guest boot/ping tests, which
we do regularly with kvm-autotest anyway.

Think of how new instruction emulations are always accompanied by new
kvm-unit-tests tests, I often don't even have to ask for them.

-- 
error compiling committee.c: too many arguments to function