Re: [Qemu-devel] [ANNOUNCE] qemu-test: a set of tests scripts for QEMU

[Blue Swirl]

On Thu, Dec 29, 2011 at 19:04, Peter Maydell <address@hidden> wrote:
> On 29 December 2011 18:35, Anthony Liguori <address@hidden> wrote:
>> On 12/29/2011 11:49 AM, Peter Maydell wrote:
>>> The next obvious question is: are we going to make a serious attempt?
>>> (For instance, in a hypothetical tests-required world, would we
>>> tell those nice folks from Samsung "no you can't land your
>>> Exynos patches unless you write 9000+ lines of test cases" ?
>>
>> The virtio-serial test case I posted was 50 lines in qemu-test.  The
>> virtio-serial driver is about ~1500 LOC.  That's about 3%.
>
> This just means it's testing only a fraction of what virtio-serial
> actually implements (as you note yourself in the comments for the
> test case). For serious coverage you'd also need to cover reading(!),
> larger quantities of data than single lines, what happens when one end
> feeds in data but the other end isn't reading, vmstate save/load,
> multiple simultaneous ports, behaviour on close-and-reopen,
> whether it works on bigendian targets, benchmarking to identify
> possible performance regressions, etc etc. I think that by the time
> you've done all that you'll be closer to 1500 lines than 50.

The coverage point of view is interesting. I'd expect the coverage
rate to rise as one moves from qtest via qemu-test to autotest.
However, I'd also expect that having qtest and qemu-test would both
increase chances that patches also get test coverage.

Gathering test coverage data could be a start.

>> I would expect that we at least have some sort of test that could verify
>> that the Exynos platform more or less worked as expected.  If that was just
>> booting a Linux kernel, that would be fine by me.
>
> Yes. There's a large range between "no tests required at all" (essentially
> what we have now) and "an equivalent level of device testing to what you
> would carry out before sending your hardware design out to be fabbed into
> silicon" (which I hope we'd all agree would be ludicrously high for QEMU);
> I'm trying to establish where we're attempting to set our bar.
>
> I agree that we'd get a lot of bang-for-the-buck out of basic automated
> "boot the guest and prod it" testing that covered most of our platforms.

That may also be too simplistic. For example, sparc-test (boot Sparc32
Linux with busybox) does not catch doubleword access problems or
certain bugs in floating point point emulation. For those, I need a
boot from an install CD which starts X.

Overall, I'd think that all of qtest, qemu-test and autotest have
their place. If one brilliant day autotest becomes so lean and mean
that qemu-test is unnecessary, it can be deprecated.