Re: [Qemu-devel] [ANNOUNCE] qemu-test: a set of tests scripts for QEMU

[Avi Kivity]

On 12/29/2011 06:12 PM, Anthony Liguori wrote:
>>> That's why I've also proposed qtest.  But having written quite a few
>>> qtest unit tests by now, you hit the limits of this type of testing
>>> pretty quickly.
>>
>> Can you describe those limits?
>
>
> I started writing a finger print test.  While it's easy to do PCI
> enumeration, it gets pretty nasty if you want to actually access BARs
> (to read virtio registers) 

Why is that? it should be pretty easy to poke values into the BAR registers.

Something else we can do is access the BARs directly.  We have APIs to
poke values into mmio, add an api to poke a value into a device's BAR. 
Now that each BAR is represented by exactly one memory region, known to
the pci core even if it is not mapped, it should be easy.

> and forget about trying to get SMBIOS information as that involves
> mucking around with ACPI.

SMBIOS is built by seabios, yes?  So fingerprinting it should be done
from seabios-test.git.  Let's divide it into two problems:

- seabios pokes qemu to get information.  We should fingerprint this
information to make sure different qemu version can interoperate with
different seabios versions (needed since we migrate the bios along with
the rest of the guest).  I'm guessing most of this info is from fwcfg? 
We can easily fingerprint it like any other device.
- we need to make sure seabios generates the same tables when using -M. 
Here, we're not testing a device, rather we're testing guest code, so it
makes sense to use a guest for this.

However, I don't think it's even necessary.  From a quick read of the
manual, SMBIOS is just a set of static tables in memory which are picked
up using a signature.  So all we need to do is boot an empty guest, read
its memory, and look for the tables.

>
> OTOH, it was very simple to write with qemu-test:
>
> http://git.qemu.org/?p=qemu-test.git;a=blob;f=tests/finger-print.sh;h=fc715378a3bbae0b458cc419981c2493d98f5c3d;hb=HEAD
>

Yes; but using Linux limits you to what it exposes (of course Linux
exposes quite a lot, so that's mostly a non issue; but we'll have
counterexamples).

>
> And I ended up finding some long standing bugs in the process:
>
> http://mid.gmane.org/address@hidden
>
>
> It's fairly nasty as it would only show up if doing migration from a
> new QEMU with a new guest to a old QEMU.

Yes, good catch.

>
> I think it's a good example of the type of test that I'm targeting at
> qemu-test.  It's really not interesting to run it across multiple
> distro types.  But doing it with qtest would be prohibitively hard.

I don't see why.  A python library to read pci config should be a couple
of dozens of lines long.  Then you iterate over all functions and print
selected registers.

>>> Please review the qtest series.  I think it offers a pretty good
>>> approach to writing this style of test.  But as I mentioned, you hit
>>> the limits pretty quickly.
>>
>> I think it's great, it looks like exactly what I wanted, except it's
>> been delivered on time.
>
>
> Can you take a look at how interrupts are handled?  From what I
> gather, the only real limitation of this approach is that we won't be
> able to simulate MSI vectors but I don't think that's a huge problem.
>
> I looked at integrating interrupt handling at CPUs itself and that
> turned out to be fairly invasive.

You mention in the changelog replacing the APIC.  Why can't you do that?

>
>> I'd really like to see it integrated quickly
>> with some flesh around it, then replying -ENOTEST to all patches.  This
>> will improve qemu's quality a lot more than guest boot/ping tests, which
>> we do regularly with kvm-autotest anyway.
>>
>> Think of how new instruction emulations are always accompanied by new
>> kvm-unit-tests tests, I often don't even have to ask for them.
>
> Yes, this is exactly where I'm heading with all of this.

It looks great.  One thing I'd change is to use the qmp protocol
(perhaps not the monitor, just the schema/codegen).  Does something
prohibit this?

-- 
error compiling committee.c: too many arguments to function