Re: [Qemu-devel] [ANNOUNCE] qemu-test: a set of tests scripts for QEMU

[Anthony Liguori]

On 12/26/2011 05:00 PM, Dor Laor wrote:
> On 12/26/2011 05:12 PM, Anthony Liguori wrote:
> > Hi Dor,
>
> Merry Christmas Anthony,

Happy Hanukkah!

> > I think it's not a bad thing to have multiple test suites when there
> > isn't considerable overlap.
>
> I agree but in this case, it loos to me that qemu-test is likely to do a subset
> of what kvm autotest do in a potentially less elegant way but due to the
> influence of a key maintainer may gain traction and dilute kvm autotest..

I'm worried that if we take an attitude of doing what we've always done, we'll 
get the same results we always have.

kvm-autotest is a great test suite, but I'm not happy about the number of 
regressions that were present in 1.0 even after a month of release candidates. 
We need to write a lot more tests than we are right now that cover a much 
broader set of functionality.

> > > > It has the following characteristics:
> > > >
> > > > 1) It builds a custom kernel and initramfs based on busybox. This is
> > > > fairly important to ensure that we can run tests with no device
> > > > pre-requisites.
> > >
> > > This can be done easily w/ autotest too.
> >
> > Okay, please demonstrate :-) The procedure to do this with qemu test is:
> >
> > $ git clone git://git.qemu.org/qemu-test.git
> > $ cd qemu-test
> > $ git submodule update --init
> > $ make
> >
> > How would one do this with kvm autotest?
>
> git clone git://github.com/autotest/autotest.git
> cd autotest/client/tests/kvm
> sudo ./get_started.py
>
> One command shorter and the above downloads F15 automatically and runs it.
>
> (from https://github.com/autotest/autotest/wiki/KVMAutotest-RunGetStartedScript)

I know how to install kvm-autotest :-)  You claimed that autotest could easily 
"build a custom kernel and initramfs based on busybox."  I've studied 
kvm-autotest extensively to determine if it could do what I want it to do and 
came to the conclusion that it couldn't without significant effort.

The infrastructure assumes that you have a full OS available in the guest.  The 
tests are written in Python and make a variety of assumptions.  To my knowledge, 
it's not very practical to build a busybox environment with Python embedded in it.

This means you need to install a distro and have a disk based install.  Not 
requiring this was one of my primary requirements in creating qemu-test mainly 
since the resulting images are very large and installing a guest with TCG is 
extremely slow.

OTOH, you can build a kernel and initrd with a cross compiler and booting a 
kernel, while not fast under TCG, is not all that slow.

> > > > 5) The tests execute very quickly, can be run stand alone, and do not
> > > > require root privileges
> > >
> > > ditto for kvm auotest. It's possible to configure it w/o root too
> > > which is not a
> > > huge issue.
> >
> > When I say, "run quickly", I mean, they execute very quickly.
>
> /me too
>
> > $ time ./qemu-test ~/build/qemu/x86_64-softmmu/qemu-system-x86_64
> > tests/virtio-serial.sh
> >
> > real 0m4.385s
> > user 0m1.460s
> > sys 0m1.860s
>
> That's impressive but it's more of a function of the guest being used - if
> instead of running a full Fedora install, you'll chose your busybox image w/
> -kernel/initrd you'll get a similar result.

I'm fairly certain that this is not easy to do with kvm-autotest.  I'm happy to 
be proven wrong but I didn't just create qemu-test in a vacuum.

> > I've used kvm-autotest a lot, there is no test in kvm-autotest that is
> > even close to completing in 4 seconds start to finish. I don't think
> > kvm-autotest can even run without installing a guest first which puts
> > puts a simple test more at the 30 minute mark. If we're talking about
> > TCG testing, then we're in hours territory.
>
> Autotest today may be too verbose by default and will call
> echo 3 > /proc/sys/vm/drop_caches but it does not have to be this way and Lucas
> can easily create a slim mode that does not even need to be root.

If it's easy, then by all means, please demonstrate.  If qemu-test only serves 
to show what my requirements for kvm-autotest are and Lucas can implement it all 
very easily, then I would be happy to discontinue qemu-test immediately.

But as I mentioned above, I don't think it's that simple.  You basically can't 
have python-based guest tests without having a full guest distribution.  That 
means in the very least, every kvm-autotest test will need to be rewritten.

> > If there's a way to run kvm-autotest as a non-privileged user, I've not
> > figured it out yet.
>
> Again, there is no magic in a test suite so if required we can wave it (maybe we
> need to do that regardless)

I cannot stress enough how important this is.  The entire process from git fetch 
to running the test should not require root privileges.  If you are targeting 
developers, you can't assume that the box is dedicated to testing as you would 
if you're doing QE testing.  That means you need to co-exist with the existing 
environment and not muck around with network setting and stuff like that.

FWIW, I gave this feedback during Lucas' talk at KVM Forum 2011 so this 
shouldn't be new news to anyone.

> > Of course, kvm-autotest can test a lot more things than qemu-test can
> > since the tests are guest agnostic. That is the primary architectural
> > difference. By writing the tests to one specific guest (busybox initrd),
> > they can be extremely fast in execution, but the tests are less general
> > purpose.
> >
> > That makes qemu-test very useful for development testing (such as long
> > automated bisect runs) but less useful for acceptance testing.
>
> Autotest too can define a family of 'sanity' or 'rapid-bisec-mode' and the tests
> will be slimmer or each test will have such a mode.

I've requested a simple config for autotest in the past and even offered to 
include it in qemu.git as a standard profile.

The simple config that I use for regression testing takes about two hours to 
complete.  I'd prefer to have something that took a few minutes to complete...

> > > Please compare your own virtio-serial test w/ the autotest version of it:
> > > https://github.com/autotest/autotest/blob/master/client/tests/kvm/tests/virtio_console.py
> > >
> > >
> > >
> > >
> > > This single file tests functionality, limits, console, live migration and
> > > performance. Of course one can add a very basic 'hello world' sanity
> > > test too
> > > that will run quickly and will identify basic breakage fast.
> >
> > Note that virtio_console.py is 2175 LOC whereas virtio-serial.sh is 52.
>
> I bet that virtio_console_super_slim_and_equal_to_qemu_unittest_serial.py will
> be less than 52 lines.

Okay, then please do demonstrate.  I don't mean to be coy here but as I 
mentioned earlier, I've studied kvm-autotest and based on my understanding of 
it, I don't think it's possible.

I'd be thrilled to be proven wrong though.

> It depends what you like to test. as noted, the above
> test fully covers all aspects of virtio-serial. It does not have to be this way
> and I do agree it is nicer to split it to multiple sub files.

Looking closely, I think the main difference is that the kvm-autotest test also 
tests guest semantics.  It tests the behavior of module unload, SIGIO, poll, etc.

While those tests are important for distro QE testing, they are not useful for 
QEMU testing as there's almost nothing that can change in QEMU that's going to 
affect that behavior.  Those are really Linux kernel tests, not QEMU tests.

> > There is a lot of value in being able to write simple tests in 50 lines
> > of code. In fact, the largest qemu-test at the moment is only ~70 lines
> > of code. And besides finding regressions in my own code (which is the
> > primary use atm), I've found and fixed a few real bugs in upstream QEMU.
>
> Now imagine what virtio_console.py buys you :)

Actually, the only thing that it tests that's a QEMU function is having multiple 
ports per PCI device.  That's something I could add to the qemu-test test with a 
couple dozen lines of code I suspect.

> > (quickly) get to a point where we can mandate that features come with
> > test cases. It is extremely hard to implement touch everything features
> > with no good way to test all of the different things we support.
> >
> > kvm autotest cannot fill that role because it's too complicated and too
> > many layers removed from QEMU.
> >
> > virtio-console/virtio-serial-bus in about 1400 LOC in QEMU. I'm not
> > about to mandate that someone writes 2.2k LOC in a python test framework
> > in order to get 1.4k of code merged in QEMU.
> >
> > But 50 lines of bash seems like a more than reasonable requirement.
>
> Hmm, what's the coverage of these 50 LOC?

It tests device creation (including via alias), device fingerprinting (although 
that's a different test), and basic I/O functionality.

> Maybe when your maintainer hat is on, it is indeed a huge task to keep all of
> qemu platforms/architectures to compile and run so that's enough but when I wear
> my Red Hat manager role, I wish that it will not only compile and boot but will
> test as much functionality as possible and will be easily re-used by R&D, QA,
> libvirt, while all supported guests are _potentially_ exercised, not just 
> busybox.

Right, there are two different use cases here:

1) light, QEMU specific testing, that can be used as a QEMU regression suite. 
Essentially, the problem to solve is that when doing changes like the memory 
API, just trying to figure out how to boot a cris guest to confirm that it 
didn't fall apart is surprisingly time consuming.  This is the kind of test 
suite that a developer would run more or less after every local commit.

2) Thorough integration tests that test the whole stack including the kernel, 
libvirt, QEMU, and guest software.  This is the sort of suite that a 
distribution would run as part of their QE procedures.

I think these goals are different enough that they deserve to be separate 
projects.  I don't think kvm-autotest would be improved by also taking on (1) as 
it would force lots of change for very little benefit to (2).

> Since kvm autotest will be there anyway, I wish that developers will contribute
> 50 LOC to autotest (and not 2.2kloc) and hope the ROI will justify it.
>
> I agree autotest is not perfect but it likes to be such.
> If you wish, you can challenge Lucas and Cleber w/ these type of requirements
> and we'll all improve as a result.

Well consider qemu-test the challenge.  It's an existence proof that we can have 
a very easy to use framework for testing that runs extremely fast and is very 
easy to write tests for.

I don't think qemu-test would ever be appropriate for distro QE testing as 
kvm-autotest does an excellent job of this.   I really think that this are two 
fundamentally different use-cases that require specialized tools.  Of course, 
I'm happy to be proven wrong.

Regards,

Anthony Liguori

> Cheers,
> Dor
>
> > Regards,
> >
> > Anthony Liguori
> >
> > > > 6) They are random by nature with the ability to fix the seed in order
> > > > to be used in git-bisect.
> > > >
> > > > I think Gerd had been looking at doing something similar with a custom
> > > > initrd.
> > > >
> > > > I've tried to consider other architectures and had hoped that we could
> > > > commit the vmlinuz and initramfs into git so that it was easy to test
> > > > other architectures without having a full build environment.
> > > > Unfortunately, busybox doesn't link statically with glibc and I can't
> > > > see an obvious way to commit binaries while respecting the GPL since we
> > > > need to pull glibc into the initramfs.
> > > >
> > > > I know buildroot exists specifically to deal with this but in my
> > > > experience, buildroot is very unreliable and extremely heavy weight
> > > > since it rebuilds gcc multiple times in order to bootstrap a ulibc
> > > > environment.
> > > >
> > > > Anyway, the code is available at:
> > > >
> > > > http://git.qemu.org/qemu-test.git
> > > >
> > > > See the README for instructions on how to use it.
> > > >
> > > > Regards,
> > > >
> > > > Anthony Liguori