Re: [Qemu-devel] Please read: make check framework

[Andreas Färber]

Am 09.01.2012 20:28, schrieb Anthony Liguori:
> On 01/09/2012 11:47 AM, Paul Brook wrote:
>> - What level of testing is appropriate? As a maintainer when can/should I
>> bounce a patch due to lack of tests?
> 
> Yes.

Note the "when". :)

>> e.g should new device emulation come with
>> unit tests? New infrastructure? What about fixes to both of the above,
>> should
>> these include regression tests?
> 
> There's a careful balance that we're going to need to work out over
> time.  We need to balance test coverage with setting reasonable criteria
> for patch inclusion.

Can we interpret that as new API infrastructure for now? As opposed to
devices (needing qtest), TCG (not unified), KVM (autotest), build
infrastructure (manually).

Do you envision also testing scripts? (e.g., simpletrace.py,
checkpatch.pl, tracetool[.py])

> We're still short on infrastructure right now so I expect that a lot of
> stuff will get merged without tests at first.  As we get more test
> infrastructure like qtest and qemu-test, I expect that to change.
> 
> In terms of total coverage for any given feature, I think ultimately the
> goal is to move the responsibility for regression testing from the
> contributors of new features to the subsystems themselves.
> 
> IOW, after we get going with our test infrastructure, when we do big
> sweeping changes like the memory API or QOM, I will have much less
> sympathy for breakages that are caused by subsystems with an incomplete
> test suite.
> 
> So I think the most important thing for maintainers to start thinking
> about is how they can add the necessary infrastructure for testing their
> subsystems.

Well, I already tried pointing out that qemu-test in the way previously
proposed does not help catching the recent regressions for PReP.

Neither am I a kernel guru to propose my own JeOS build and testing
framework, nor do I think an abundance of test frameworks merely
orchestrated by a common driver will do us much good...

>From what I understand, qemu-test with building binutils, gcc, linux
would not fit the stated 10 minutes goal.

The intentions here are certainly good, just the process needs some more
thought - or less absoluteness in announcing it. If the infrastructure
we provide in a first step is good and people benefit from using it, we
will gain coverage and supporting infrastructure over time. It's
unlikely to work as a requirement from day one though.

Regards,
Andreas