|
From: | Anthony Liguori |
Subject: | Re: [Qemu-devel] [ANNOUNCE] qemu-test: a set of tests scripts for QEMU |
Date: | Thu, 29 Dec 2011 11:02:35 -0600 |
User-agent: | Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.23) Gecko/20110922 Lightning/1.0b2 Thunderbird/3.1.15 |
On 12/29/2011 10:53 AM, Avi Kivity wrote:
On 12/29/2011 06:39 PM, Anthony Liguori wrote:qemu-test builds a custom guest, entirely from source. This makes it possible to efficiently test non-native architectures. The tests are written for this minimal environment (which is not straight forward to do). This tests will never run on Windows as they are 100% tied to their environment. autotest (let's be clear, there is no such thing as kvm autotest anymore) is a framework for executing third party tests. It's got fancy magic to execute in client mode vs. server mode. There is a suite of tests that are integrated in autotest that exercise various types of virtualization functionality. This suite of tests use a special config format to determine what they do. Included in this, is the ability to create a guest from an ISO either using step files or via a guest-specific mechanism (kickstart, etc.). The tests are written to be mostly guest agnostic and are therefore written in Python in a cross platform way. There is essentially no overlap between the two and calling kvm autotest superior is like calling the Sun superior to the Moon.Why not extend autotest do the same thing. Instead of downloading a fedora iso it would download a kernel tarball and (cross-)build it.
My plan is to post a qemu-jeos repo that has the build bits for creating the kernel/initramfs pair.
qemu-test will then live in qemu.git and just depend on the results of the qemu-jeos.git. I'd maybe even go as far as including the binaries in qemu.git but I don't think that would work so well...
My main motivation to merge qemu-test and kvm autotest is that I fear that qemu-test will be the only requirement for committing stuff for qemu and developers will be excused from their 'duty' by writing a 50 LOC shell script and assume they done w/ testing.There is no requirement to write autotest tests to get things merged into QEMU. That's is how things are today. And I don't think there will ever a requirement to write anything that isn't merged directly into qemu.git. I'm not going to hold up merging a feature until another project merges something into their tree.What about virtio features (we used to depend on the kernel, now on the spec)? Seabios?
Those are rare occurrences and have always been a bit awkward. OTOH, we're talking about the critical path of essentially every single feature that gets merged into QEMU.
So unless we're going to merge KVM autotest into qemu.git, I don't think there's every going to be a requirement to write a KVM autotest test in order to get something merged into qemu.git.Let's consider postcopy live migration being proposed. In addition to various unit tests, it also wants an integration test. So now we need to write a qemu-test postcopy live migration test, and also autotest test that tests non-Linux guests, migration under load, with memory hotplug, etc.
But the integration test is also going to depend on libvirt support for the feature. So how does this all get orchestrated? kvm-autotest merges the test case first, then QEMU merges the support, then libvirt? What about VDSM and ovirt-engine? Once kvm-autotest starts supporting VDSM that's yet another component to deal with.
Realistically, kvm-autotest is going to have to wait for the bits to get merged in all of the necessary pieces and then build a test for it on their own schedule.
But since autotest is a framework for running third-party tests, it seems to make sense for qemu.git to have a test framework that autotest can execute. And since what we call KVM autotest actually tests a heck of a lot more than just QEMU, it makes sense for KVM autotest to continue to focus on full stack testing where QEMU is but one of the many components that it tests.It might have made sense to split the kvm-testing functionality of autotest, and have autotest drive that. We could even have called it qemu-test.
I specifically advocated this during Lucas' KVM Forum talk and he was strongly opposed to it.
I think kvm autotest would get a lot more interest if the test cases were pulled out of autotest, made more stand alone. They also should be more Unix like being divided into individual executables with independent command line options over a single do-everything configuration file.
But that's all independent of qemu-test. qemu-test is really qtest + Linux/busybox as a libOS. Even if the above happened, I would still think qemu-test needed to exist.
Regards, Anthony Liguori
[Prev in Thread] | Current Thread | [Next in Thread] |