Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests

[Anthony Liguori]

On 03/09/2012 06:48 AM, Lucas Meneghel Rodrigues wrote:
> On 03/09/2012 09:13 AM, Anthony Liguori wrote:
> > On 03/08/2012 05:07 PM, Lucas Meneghel Rodrigues wrote:
> > > Here is the qemu-test version
> > >
> > > http://git.qemu.org/?p=qemu-test.git;a=blob;f=tests/virtio-serial.sh;h=e95ae6e0b63758262919702d51a9c83bebe2fb08;hb=master
> >
> > So virtio-serial is an exception in autotest:
> >
> > 2174 virtio_console.py
> > 1875 cgroup.py
> > 615 ksm_overcommit.py
> > 439 qemu_img.py
> > 407 qmp_basic.py
> > 389 qmp_basic_rhel6.py
> > 356 stepmaker.py
> > 247 steps.py
> >
> > // The next largest actual test of QEMU
> > 203 pci_hotplug.py
> > 190 cdrom.py
> > 182 physical_resources_check.py
> > 181 timedrift.py
> > 170 enospc.py
> > 150 balloon_check.py
> > 138 multi_disk.py
> > 121 unittest.py
> > 117 migration.py
> > 111 cpu_hotplug.py
> > 107 migration_multi_host.py
> > 104 nic_hotplug.py
> > 103 timedrift_with_stop.py
> > 96 timedrift_with_migration.py
> > ...
> >
> > So picking virtio-serial as your comparison point is not really
> > representative of kvm-autotest but at any rate...
>
> We have a bunch of high level test functions in client/virt/virt_test_utils.py
> that contain some commonly used test functions such as migration and running
> autotest tests on vms, and other functions, that allow us to reuse those
> functions on the tests and save code, but we can reasonably assume that it
> doesn't change the order of magnitude of the actual qemu tests in size, so point
> taken.
>
> You also have a point in the respect that a lot of the large tests are more
> linux-qemu integration tests, name cpuflags, cgroups, ksm_overcommit.

And this is exactly the type of thing that autotest will always be the best tool 
for.

> The point you tried to make and I replied to was 'qemu-test tests are all
> smaller than the equivalent kvm autotest tests'. Well, sure they tend to be, but
> in pretty much all cases more code means more functionality being covered, and
> making sure the same test works on RHEL5, RHEL6, upstream, on an Ubuntu, Fedora,
> RHEL or even Windows guests.

But this is not the scope of qemu-test.  And we may end up duplicating some 
things here but does it really matter?  We're talking about a few dozen python 
scripts that are a ~100 lines.

> It is indeed a bit nerve wrecking to hear that all you can do with the stuff you
> have been working on the last 3 years can be done better with a dozen of shell
> script functions. It's similar to say that we just like to throw lines at a text
> editor just for the fun of it. I am sure you didn't mean it but that is how it
> sounded, and that's why I'd like to assure that the code there *does stuff*.

Look at how this discussion started.  We've been discussing testing on 
qemu-devel at excruciating length and detail and have finally come to something 
of a consensus.  AFAIK, no one from autotest has participated in those 
discussions which is fair as I'm sure ya'll don't read qemu-devel religiously.

Then we see this note that more or less declares, this is how QEMU should do all 
of its testing.  What reaction did you really expect there to be? :-)

> It's just that this extra stuff is potentially not interesting to the goal of
> doing developer level regression testing of qemu alone.

I think we need to focus this discussion on concrete technical proposals.  If 
the proposal is, QEMU should use libautotest, we need to start with an awful lot 
more detail about libautotest does and what functions it provides.

Regards,

Anthony Liguori