qemu-devel
[Top][All Lists]
Advanced

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Qemu-devel] [ANNOUNCE] qemu-test: a set of tests scripts for QEMU


From: Anthony Liguori
Subject: Re: [Qemu-devel] [ANNOUNCE] qemu-test: a set of tests scripts for QEMU
Date: Wed, 28 Dec 2011 08:27:48 -0600
User-agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.23) Gecko/20110922 Lightning/1.0b2 Thunderbird/3.1.15

On 12/27/2011 11:01 PM, Cleber Rosa wrote:
On 12/27/2011 11:37 PM, Anthony Liguori wrote:
I think the main goal of qemu-tests (may be implicit) is to be quick and simple.

qemu-test doesn't have a main goal. My goal is to improve QEMU's quality.
qemu-test is just a tool to help achieve that goal.

Maybe I've used the wrong wording. I got the feeling that, besides testing qemu
the way you need it, keeping qemu-test simple was really important.

Simple is always important. In the case of qemu-test, there are some important trade offs made to keep things simple and fast. It doesn't try to work with arbitrary guests which means the tests need to handle only one environment. The guest is pre-made to have exactly what is needed for the tests so there is no setup required. The guest is as small as possible such that the test can run as quickly as possible.

That is indeed great, but if one thinks that's all we'll ever going to need,
that thought is pretty naive.

I don't know who "we" is, but I can tell you that qemu-test is exactly what
*I* need. Consider that I spent a good portion of every single day testing
QEMU with either my own or other people's patches, making that job easier and
more automated is fairly important to me.

"We" is everyone that somehow contributes to QEMU, that is, the QEMU community.
If you're only concerned about what *you* need, then you're on the right track.
If, besides that, you feel it'd be nice to *try to* concentrate our efforts,
then we're all on the same track.

There is no need to have a single tool that meets every possible need. In fact, the Unix tradition is to have separate single purposed tools.

Having two test tools it not a bad thing provided that the overlap isn't significant. We shouldn't be discussing whether it's possible to merge the two tools, but rather what the technical benefits of doing so would be.

Since at this point, there is almost no overlap between the two, I don't see any actual technical benefit to merging them. I see benefit to autotest executing qemu-test, of course.

I'm sharing it because I suspect that a lot of other developers have a similar
need.

And it may be true that there's room for both test
suites... or that, as busy developers, we're refusing to deal with the added
complexity (qemu alone accounts for a lot) and delaying to fix the fixable. I
believe on the later.

One example: kvm-autotest has a complex configuration file format with a steep
learning curve, while a test such as qemu-tests/tests/simple-ping.sh would have
to be tweaked if somehow the kernel detects the first ethernet interface as em1
(such as recent Fedora systems do). Simple, but not scalable.

I can tell by this comment that you don't actually understand how qemu-test
works. Please take a look at it before jumping to conclusions about whether it
should or shouldn't be part of kvm-autotest.

Hint: qemu-test always uses the same kernel because it builds it as part of
the test suite. The behavior of how a nic test will never change unless
someone explicitly changes the kernel.

I can tell you did not get my point: I'm giving some reasons of why current kvm
autotest is somehow complex, and how qemu-tests gets away and keeps it simple.

You're claiming that "we're refusing to deal with the added complexity (qemu alone accounts for a lot) and delaying to fix the fixable".

There is no way that qemu-test would ever need to deal with how Fedora configures udev to name ethernet devices without becoming something totally different than it is. So there's no "delaying to fix the fixable" here.

qemu-test makes a simplifying assumption. By restricting the guest to a fixed environment (initramfs w/busybox), things inherently become much, much simpler.

Of course, this is not an adequate assumption to make if it were our only test tool but fortunately, we have an existing test suite that does a very good job at testing a wide variety of guests :-)


The Python requirement inside the guest is true *if* we want to run regular
autotest tests inside the guest (see autotest/client/virt/tests/autotest.py) and
this accounts for very very little of kvm autotest usage. All other tests
interact with the monitor directly and with the guest via ssh/telnet/serial.

qemu-test does not require any specific hardware to be used in the guest which
lets it test a wider variety of scenarios in QEMU. So you cannot assume there
is ssh/telnet/serial available.

I really thought we could rely on, at least, a serial connection. If not, then
indeed the current kvm autotest approach is not compatible with that test
environment. That is not to say that kvm autotest couldn't incorporate the
qemu-tests approach/functionality.

With the scope that qemu-test has, it cannot assume any hardware is present because it wants to test every piece of hardware.

BTW, I just don't like the idea of having lots of functionalities/tests
implemented on two test suites for a single piece of software, unless proven
that there's no way around it. To me, this is the whole point of this 
discussion.

I actually disagree on principle. Two tools shouldn't be combined unless there's a proven reason that they benefit from being together. You get more flexibility at the end of the day having strong, independent components that can be combined together.

So, I see no reason for not using a more expressive language,

I seriously doubt you can build a useful initramfs that contains python
without doing something crazy like livecd-tools does....

You're right. Again, I was thinking we could rely at least on a serial
connection. Can we not?

No.  How would you test creating a guest with no serial devices?

The nice thing about delivering a test via the initramfs is that you're only injecting software into the guest (usually via some non guest visible mechanism).

I don't see any reason why everything needs to live in kvm-autotest... but if
you really feel that way, please provide patches that demonstrate how this
would work.

If it's technically viable, I think that having it as part of kvm autotest,
shows that the project is more cohesive, attracts more contributions, and makes
better use of our efforts.

Just putting the code in kvm-autotest.git in a directory doesn't make sense to me. Beyond the lack of a technical reason to do so, logistically, it makes it harder for me to ask people to submit test cases with a patch series if I can't actually apply the test case when I'm applying the patches to qemu.git.

If qemu-test didn't use large submodules (like linux.git), I would have made qemu-test part of qemu.git. As far as I'm concerned, qemu-test.git is just an extension to qemu.git.

I think there's something of a knee jerk reaction here. The existence of
qemu-test does not take anything away from kvm-autotest. It's just another
tool in our arsenal to achieve our joint goal of making QEMU (and KVM) higher
quality.

You're right, It does not take anything away from kvm autotest today. But
suppose we can prove that kvm autotest can indeed absorve all of qemu-tests
functionalities, it'd be itself a reason for doing so.

That's not obvious to me.

It'd avoid finding
ourselves with two evolved test tools that do some of the same things, but are
separate implementations.

It's just too easy to argue about abstract things like this. This discussion about testing has come up many times and we've talked to death how much we need to do better testing and get to test driven development.

Yet, not much has materialized. I'm not interested in talking about things in abstract anymore. That's why I announced qemu-test. If folks think it should behave differently, then patches are welcome.

If you think that kvm-autotest can do everything that qemu-test does, just make the changes to kvm-autotest to make it behave the same way and show me. If it makes sense, I'll happily embrace it.

Regards,

Anthony Liguori



reply via email to

[Prev in Thread] Current Thread [Next in Thread]