Re: [Qemu-devel] [ANNOUNCE] qemu-test: a set of tests scripts for QEMU

[Cleber Rosa]

On 12/26/2011 08:00 PM, Dor Laor wrote:
> On 12/26/2011 05:12 PM, Anthony Liguori wrote:
> > Hi Dor,
>
> Merry Christmas Anthony,
>
> > On 12/25/2011 09:19 AM, Dor Laor wrote:
> > > On 12/19/2011 07:13 PM, Anthony Liguori wrote:
> > >
> > > Well, I'm still not convinced that a new standalone package should
> > > handle these
> > > cases instead of kvm autotest. I'll be happy to integrate the tests to
> > > kvm
> > > autotest anyway and the more the merrier but imho it's a duplicate.
> >
> > I'm sure kvm autotest could be taught to do exactly what qemu-test is
> > doing. But why does kvm autotest have to do everything? I doubt there
> > would be much code reuse.
> >
> > I think it's not a bad thing to have multiple test suites when there
> > isn't considerable overlap.

I think the main goal of qemu-tests (may be implicit) is to be quick and 
simple. That is indeed great, but if one thinks that's all we'll ever 
going to need, that thought is pretty naive. And it may be true that 
there's room for both test suites... or that, as busy developers, we're 
refusing to deal with the added complexity (qemu alone accounts for a 
lot) and delaying to fix the fixable. I believe on the later.

One example: kvm-autotest has a complex configuration file format with a 
steep learning curve, while a test such as 
qemu-tests/tests/simple-ping.sh would have to be tweaked if somehow the 
kernel detects the first ethernet interface as em1 (such as recent 
Fedora systems do). Simple, but not scalable.

Other example: kvm-autotest can easily implement (copy?) the 
functionality to build a busybox initramfs. We already have code to 
build from sources such as git, local source directories, remote 
tarballs, you named it. And the recipe to build a given package is also 
configurable/extensible so we could even leverage that. qemu-tests, on 
the other hand, would need lots of code to test the same range of guests 
that kvm-autotest tests. Simple, but limited.

So, it looks like, if we add the missing features to both test suites, 
we'll end up with two implementations, using different 
languages/approaches, but with very little added value.

> I agree but in this case, it loos to me that qemu-test is likely to do 
> a subset of what kvm autotest do in a potentially less elegant way but 
> due to the influence of a key maintainer may gain traction and dilute 
> kvm autotest..
>
> > > > It has the following characteristics:
> > > >
> > > > 1) It builds a custom kernel and initramfs based on busybox. This is
> > > > fairly important to ensure that we can run tests with no device
> > > > pre-requisites.
> > >
> > > This can be done easily w/ autotest too.

The Python requirement inside the guest is true *if* we want to run 
regular autotest tests inside the guest (see 
autotest/client/virt/tests/autotest.py) and this accounts for very very 
little of kvm autotest usage. All other tests interact with the monitor 
directly and with the guest via ssh/telnet/serial.

So, I see no reason for not using a more expressive language, having an 
API, etc, to do that's done in in_host(). In the guest side, what 
changes is that we send one command at a time, checking/collecting 
results, instead of pushing the complete tests files into the initramfs.

> > Okay, please demonstrate :-) The procedure to do this with qemu test is:
> >
> > $ git clone git://git.qemu.org/qemu-test.git
> > $ cd qemu-test
> > $ git submodule update --init
> > $ make
> >
> > How would one do this with kvm autotest?
>
> git clone git://github.com/autotest/autotest.git
> cd autotest/client/tests/kvm
> sudo ./get_started.py
>
> One command shorter and the above downloads F15 automatically and runs 
> it.
>
> (from 
> https://github.com/autotest/autotest/wiki/KVMAutotest-RunGetStartedScript)
>
> > > > 2) Tests are scripts that are launched in the initramfs
> > > >
> > > > 3) The test controls exactly how QEMU is launched which allows easy
> > > > testing of various QEMU options
> > >
> > > afaik kvm autotest do the same. It's true that kvm autotest might look
> > > less
> > > friendly but its may only be this way since it has lots of options.
> >
> > Actually, kvm-autotest has various layers of abstraction in how QEMU
> > ends up being launched. As you mention below, those layers are there to
> > allow for things like using libvirt.

Indeed the qemu command line parameters gets generated depending on many 
configuration parameters. It'd be *really* simple to add a configuration 
parameters that overwrites the qemu command with an static one.

> It goes beyond that, since it also related to the monitor interface as 
> well.
>
> > That's desirable when you're doing "virt testing", but not so desirably
> > when you're trying to write specific unit tests against QEMU.
>
> True, one may not need it at all but it's nice that a test for 
> migration/stress/hotplug will be tested directly w/ qemu and libvirt 
> w/ the same effort.
>
> > > > 5) The tests execute very quickly, can be run stand alone, and do not
> > > > require root privileges
> > >
> > > ditto for kvm auotest. It's possible to configure it w/o root too
> > > which is not a
> > > huge issue.
> >
> > When I say, "run quickly", I mean, they execute very quickly.
>
> /me too
>
> > $ time ./qemu-test ~/build/qemu/x86_64-softmmu/qemu-system-x86_64
> > tests/virtio-serial.sh
> >
> > real 0m4.385s
> > user 0m1.460s
> > sys 0m1.860s
>
> That's impressive but it's more of a function of the guest being used 
> - if instead of running a full Fedora install, you'll chose your 
> busybox image w/ -kernel/initrd you'll get a similar result.

I also think so. Maybe kvm-autotest would take a little more time 
because of the different approach we take when communicating with the 
guest, but I bet it'd be irrelevant.

> > I've used kvm-autotest a lot, there is no test in kvm-autotest that is
> > even close to completing in 4 seconds start to finish. I don't think
> > kvm-autotest can even run without installing a guest first which puts
> > puts a simple test more at the 30 minute mark. If we're talking about
> > TCG testing, then we're in hours territory.
>
> Autotest today may be too verbose by default and will call
> echo 3 > /proc/sys/vm/drop_caches but it does not have to be this way 
> and Lucas can easily create a slim mode that does not even need to be 
> root.

The drop_caches thing has been configurable for ages.

> > If there's a way to run kvm-autotest as a non-privileged user, I've not
> > figured it out yet.
>
> Again, there is no magic in a test suite so if required we can wave it 
> (maybe we need to do that regardless)

Some of the things we do in kvm autotest (and are not currently done in 
qemu-tests) indeed need root privileges, but we could add granularity to 
that.

> > Of course, kvm-autotest can test a lot more things than qemu-test can
> > since the tests are guest agnostic. That is the primary architectural
> > difference. By writing the tests to one specific guest (busybox initrd),
> > they can be extremely fast in execution, but the tests are less general
> > purpose.
> >
> > That makes qemu-test very useful for development testing (such as long
> > automated bisect runs) but less useful for acceptance testing. 
>
>
> Autotest too can define a family of 'sanity' or 'rapid-bisec-mode' and 
> the tests will be slimmer or each test will have such a mode.
>
> > > Please compare your own virtio-serial test w/ the autotest version 
> > > of it:
> > > https://github.com/autotest/autotest/blob/master/client/tests/kvm/tests/virtio_console.py 
> > >
> > >
> > >
> > >
> > > This single file tests functionality, limits, console, live 
> > > migration and
> > > performance. Of course one can add a very basic 'hello world' sanity
> > > test too
> > > that will run quickly and will identify basic breakage fast.
> >
> > Note that virtio_console.py is 2175 LOC whereas virtio-serial.sh is 52.
>
> I bet that 
> virtio_console_super_slim_and_equal_to_qemu_unittest_serial.py will be 
> less than 52 lines. It depends what you like to test. as noted, the 
> above test fully covers all aspects of virtio-serial. It does not have 
> to be this way and I do agree it is nicer to split it to multiple sub 
> files.
>
> > There is a lot of value in being able to write simple tests in 50 lines
> > of code. In fact, the largest qemu-test at the moment is only ~70 lines
> > of code. And besides finding regressions in my own code (which is the
> > primary use atm), I've found and fixed a few real bugs in upstream QEMU.
>
> Now imagine what virtio_console.py buys you :)
>
> > For a couple hundred lines of bash script, how can it be anything but a
> > good thing :-)
> >
> > > Noways we abstract kvm autotest so libvirt will be optionally tested
> > > too w/ the
> > > same tests.
> > >
> > > Again, I'm not advocating that kvm autotest is a solution for anything
> > > but for
> > > plain guest-host communication, monitor commands, etc it's a really
> > > good tool.
> > >
> > > I agree that kvm autotest may be less friendly for developer users
> > > since it
> > > carries allot of options for testing a huge matrix. Lucas and Cleber
> > > are working
> > > these days to add a make kvmautotest target to qemu so one would be
> > > able to
> > > quickly execute autotest and we can think of additional parameters like
> > > sanity-set, migration-set, etc.
> >
> > We (the QEMU project) need to get more serious about testing. We need to
>
> Like
>
> > (quickly) get to a point where we can mandate that features come with
> > test cases. It is extremely hard to implement touch everything features
> > with no good way to test all of the different things we support.
> >
> > kvm autotest cannot fill that role because it's too complicated and too
> > many layers removed from QEMU.
> >
> > virtio-console/virtio-serial-bus in about 1400 LOC in QEMU. I'm not
> > about to mandate that someone writes 2.2k LOC in a python test framework
> > in order to get 1.4k of code merged in QEMU.
> >
> > But 50 lines of bash seems like a more than reasonable requirement.
>
> Hmm, what's the coverage of these 50 LOC?
> Maybe when your maintainer hat is on, it is indeed a huge task to keep 
> all of qemu platforms/architectures to compile and run so that's 
> enough but when I wear my Red Hat manager role, I wish that it will 
> not only compile and boot but will test as much functionality as 
> possible and will be easily re-used by R&D, QA, libvirt, while all 
> supported guests are _potentially_ exercised, not just busybox.
>
> Since kvm autotest will be there anyway, I wish that developers will 
> contribute 50 LOC to autotest (and not 2.2kloc) and hope the ROI will 
> justify it.
>
> I agree autotest is not perfect but it likes to be such.
> If you wish, you can challenge Lucas and Cleber w/ these type of 
> requirements and we'll all improve as a result.

Yes, I believe it'd be a nice exercise for all of us.

The only thing I ask is that we bear at least with some of the 
complexity that kvm-autotest inherently holds... the last requests we 
had was to get rid of all the complexity, while retaining all the other 
nice characteristics. Pretty hard, so I think we failed, or maybe only 
half-succeeded at it.

> Cheers,
> Dor
>
> > Regards,
> >
> > Anthony Liguori
> >
> > > > 6) They are random by nature with the ability to fix the seed in order
> > > > to be used in git-bisect.
> > > >
> > > > I think Gerd had been looking at doing something similar with a custom
> > > > initrd.
> > > >
> > > > I've tried to consider other architectures and had hoped that we could
> > > > commit the vmlinuz and initramfs into git so that it was easy to test
> > > > other architectures without having a full build environment.
> > > > Unfortunately, busybox doesn't link statically with glibc and I can't
> > > > see an obvious way to commit binaries while respecting the GPL 
> > > > since we
> > > > need to pull glibc into the initramfs.
> > > >
> > > > I know buildroot exists specifically to deal with this but in my
> > > > experience, buildroot is very unreliable and extremely heavy weight
> > > > since it rebuilds gcc multiple times in order to bootstrap a ulibc
> > > > environment.
> > > >
> > > > Anyway, the code is available at:
> > > >
> > > > http://git.qemu.org/qemu-test.git
> > > >
> > > > See the README for instructions on how to use it.
> > > >
> > > > Regards,
> > > >
> > > > Anthony Liguori