Re: [Qemu-devel] [ANNOUNCE] qemu-test: a set of tests scripts for QEMU

[Anthony Liguori]

On 12/28/2011 11:26 AM, Avi Kivity wrote:
> On 12/28/2011 06:44 PM, Anthony Liguori wrote:
> > On 12/28/2011 09:28 AM, Avi Kivity wrote:
> > > On 12/28/2011 05:01 PM, Avi Kivity wrote:
> > > > I'd say that running a ping test is a weak version of kvm-autotest's
> > > > system tests.  Running a synthetic test that pokes values into memory
> > > > and mmio and sees a packet coming out is a unit test (the latter can in
> > > > fact be executed without a guest at all, just a table driving calls to
> > > > the memory and irq APIs).
> > >
> > > Consider
> > >     98d23704138e0
> > >     7b4252e83f6f7d
> > >     f7e80adf3cc4
> > >     c16ada980f43
> > >     4abf12f4ea8
> > >
> > > (found by looking for 'fix' in the commit log and filtering out the
> > > commits that don't support my case)
> > >
> > > how can you reject such patches on the grounds that they're not
> > > accompanied by unit tests?
> >
> > That's why I've also proposed qtest.  But having written quite a few
> > qtest unit tests by now, you hit the limits of this type of testing
> > pretty quickly.
>
> Can you describe those limits?

I started writing a finger print test.  While it's easy to do PCI enumeration, 
it gets pretty nasty if you want to actually access BARs (to read virtio 
registers) and forget about trying to get SMBIOS information as that involves 
mucking around with ACPI.

OTOH, it was very simple to write with qemu-test:

http://git.qemu.org/?p=qemu-test.git;a=blob;f=tests/finger-print.sh;h=fc715378a3bbae0b458cc419981c2493d98f5c3d;hb=HEAD

And I ended up finding some long standing bugs in the process:

http://mid.gmane.org/address@hidden

It's fairly nasty as it would only show up if doing migration from a new QEMU 
with a new guest to a old QEMU.

I think it's a good example of the type of test that I'm targeting at qemu-test. 
 It's really not interesting to run it across multiple distro types.  But doing 
it with qtest would be prohibitively hard.

> > > only by making it easy to add tests for
> > > them.  I think it would be hard/impossible to test them with
> > > linux-as-a-guest, since they fix edge cases that linux doesn't invoke.
> > > But by having our own driver (often just using qmp to poke at memory),
> > > we can easily generate the sequence that triggers the error.
> > >
> > > We'd probably need a library to support setting up a pci device's BARs,
> > > but that's easy with qmp/python integration.  You can even poke a small
> > > executable into memory and execute it directly, if you really need guest
> > > cpu interaction.
> >
> > Please review the qtest series.  I think it offers a pretty good
> > approach to writing this style of test.  But as I mentioned, you hit
> > the limits pretty quickly.
>
> I think it's great, it looks like exactly what I wanted, except it's
> been delivered on time.

Can you take a look at how interrupts are handled?  From what I gather, the only 
real limitation of this approach is that we won't be able to simulate MSI 
vectors but I don't think that's a huge problem.

I looked at integrating interrupt handling at CPUs itself and that turned out to 
be fairly invasive.

> I'd really like to see it integrated quickly
> with some flesh around it, then replying -ENOTEST to all patches.  This
> will improve qemu's quality a lot more than guest boot/ping tests, which
> we do regularly with kvm-autotest anyway.
>
> Think of how new instruction emulations are always accompanied by new
> kvm-unit-tests tests, I often don't even have to ask for them.

Yes, this is exactly where I'm heading with all of this.

Regards,

Anthony Liguori