Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests

[Lucas Meneghel Rodrigues]

On 03/09/2012 09:13 AM, Anthony Liguori wrote:
> On 03/08/2012 05:07 PM, Lucas Meneghel Rodrigues wrote:
> > Here is the qemu-test version
> >
> > http://git.qemu.org/?p=qemu-test.git;a=blob;f=tests/virtio-serial.sh;h=e95ae6e0b63758262919702d51a9c83bebe2fb08;hb=master
>
> So virtio-serial is an exception in autotest:
>
> 2174 virtio_console.py
> 1875 cgroup.py
> 615 ksm_overcommit.py
> 439 qemu_img.py
> 407 qmp_basic.py
> 389 qmp_basic_rhel6.py
> 356 stepmaker.py
> 247 steps.py
>
> // The next largest actual test of QEMU
> 203 pci_hotplug.py
> 190 cdrom.py
> 182 physical_resources_check.py
> 181 timedrift.py
> 170 enospc.py
> 150 balloon_check.py
> 138 multi_disk.py
> 121 unittest.py
> 117 migration.py
> 111 cpu_hotplug.py
> 107 migration_multi_host.py
> 104 nic_hotplug.py
> 103 timedrift_with_stop.py
> 96 timedrift_with_migration.py
> ...
>
> So picking virtio-serial as your comparison point is not really
> representative of kvm-autotest but at any rate...

We have a bunch of high level test functions in 
client/virt/virt_test_utils.py that contain some commonly used test 
functions such as migration and running autotest tests on vms, and other 
functions, that allow us to reuse those functions on the tests and save 
code, but we can reasonably assume that it doesn't change the order of 
magnitude of the actual qemu tests in size, so point taken.

You also have a point in the respect that a lot of the large tests are 
more linux-qemu integration tests, name cpuflags, cgroups, ksm_overcommit.

The point you tried to make and I replied to was 'qemu-test tests are 
all smaller than the equivalent kvm autotest tests'. Well, sure they 
tend to be, but in pretty much all cases more code means more 
functionality being covered, and making sure the same test works on 
RHEL5, RHEL6, upstream, on an Ubuntu, Fedora, RHEL or even Windows guests.

It is indeed a bit nerve wrecking to hear that all you can do with the 
stuff you have been working on the last 3 years can be done better with 
a dozen of shell script functions. It's similar to say that we just like 
to throw lines at a text editor just for the fun of it. I am sure you 
didn't mean it but that is how it sounded, and that's why I'd like to 
assure that the code there *does stuff*. It's just that this extra stuff 
is potentially not interesting to the goal of doing developer level 
regression testing of qemu alone.