qemu-devel
[Top][All Lists]
Advanced

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Qemu-devel] [ANNOUNCE] qemu-test: a set of tests scripts for QEMU


From: Anthony Liguori
Subject: Re: [Qemu-devel] [ANNOUNCE] qemu-test: a set of tests scripts for QEMU
Date: Thu, 29 Dec 2011 11:02:35 -0600
User-agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.23) Gecko/20110922 Lightning/1.0b2 Thunderbird/3.1.15

On 12/29/2011 10:53 AM, Avi Kivity wrote:
On 12/29/2011 06:39 PM, Anthony Liguori wrote:

qemu-test builds a custom guest, entirely from source.  This makes it
possible to efficiently test non-native architectures.  The tests are
written for this minimal environment (which is not straight forward to
do).  This tests will never run on Windows as they are 100% tied to
their environment.

autotest (let's be clear, there is no such thing as kvm autotest
anymore) is a framework for executing third party tests.  It's got
fancy magic to execute in client mode vs. server mode.  There is a
suite of tests that are integrated in autotest that exercise various
types of virtualization functionality.  This suite of tests use a
special config format to determine what they do.

Included in this, is the ability to create a guest from an ISO either
using step files or via a guest-specific mechanism (kickstart, etc.).
The tests are written to be mostly guest agnostic and are therefore
written in Python in a cross platform way.

There is essentially no overlap between the two and calling kvm
autotest superior is like calling the Sun superior to the Moon.

Why not extend autotest do the same thing.  Instead of downloading a
fedora iso it would download a kernel tarball and (cross-)build it.

My plan is to post a qemu-jeos repo that has the build bits for creating the kernel/initramfs pair.

qemu-test will then live in qemu.git and just depend on the results of the qemu-jeos.git. I'd maybe even go as far as including the binaries in qemu.git but I don't think that would work so well...

My main motivation to merge
qemu-test and kvm autotest is that I fear that qemu-test will be the
only
requirement for committing stuff for qemu and developers will be
excused from
their 'duty' by writing a 50 LOC shell script and assume they done w/
testing.

There is no requirement to write autotest tests to get things merged
into QEMU.  That's is how things are today.

And I don't think there will ever a requirement to write anything that
isn't merged directly into qemu.git.  I'm not going to hold up merging
a feature until another project merges something into their tree.

What about virtio features (we used to depend on the kernel, now on the
spec)?  Seabios?

Those are rare occurrences and have always been a bit awkward. OTOH, we're talking about the critical path of essentially every single feature that gets merged into QEMU.

So unless we're going to merge KVM autotest into qemu.git, I don't
think there's every going to be a requirement to write a KVM autotest
test in order to get something merged into qemu.git.

Let's consider postcopy live migration being proposed.  In addition to
various unit tests, it also wants an integration test.  So now we need
to write a qemu-test postcopy live migration test, and also autotest
test that tests non-Linux guests, migration under load, with memory
hotplug, etc.

But the integration test is also going to depend on libvirt support for the feature. So how does this all get orchestrated? kvm-autotest merges the test case first, then QEMU merges the support, then libvirt? What about VDSM and ovirt-engine? Once kvm-autotest starts supporting VDSM that's yet another component to deal with.

Realistically, kvm-autotest is going to have to wait for the bits to get merged in all of the necessary pieces and then build a test for it on their own schedule.

But since autotest is a framework for running third-party tests, it
seems to make sense for qemu.git to have a test framework that
autotest can execute.

And since what we call KVM autotest actually tests a heck of a lot
more than just QEMU, it makes sense for KVM autotest to continue to
focus on full stack testing where QEMU is but one of the many
components that it tests.

It might have made sense to split the kvm-testing functionality of
autotest, and have autotest drive that.  We could even have called it
qemu-test.

I specifically advocated this during Lucas' KVM Forum talk and he was strongly opposed to it.

I think kvm autotest would get a lot more interest if the test cases were pulled out of autotest, made more stand alone. They also should be more Unix like being divided into individual executables with independent command line options over a single do-everything configuration file.

But that's all independent of qemu-test. qemu-test is really qtest + Linux/busybox as a libOS. Even if the above happened, I would still think qemu-test needed to exist.

Regards,

Anthony Liguori





reply via email to

[Prev in Thread] Current Thread [Next in Thread]