Re: [Qemu-devel] [ANNOUNCE] qemu-test: a set of tests scripts for QEMU

[Anthony Liguori]

On 12/28/2011 11:21 AM, Avi Kivity wrote:
> On 12/28/2011 06:42 PM, Anthony Liguori wrote:
> > > In fact using linux as a guest negates that.  First of all, which linux
> > > version? if it's fixed, you'll eventually miss functionality and need to
> > > migrate.  If it keeps changing, so does your test, and it will keep
> > > breaking.
> >
> >
> > The kernel is a git submodule so it's a very specific version.  Yes,
> > we may need to bump the version down the road and obviously, if we
> > have to change any tests in the process, we can.
>
> Having a full linux source as part of the build process detracts
> somewhat from the advantages here.

Except that binaries can be made available.  The initramfs is currently 512k 
while the kernel is about 5MB.

OTOH, a Fedora install runs in the multiple GB.  Having kernels/initramfs for 
the dozen target types that QEMU supports is reasonable.  Having Fedora installs 
for all of them probably isn't for most people.

> > But there's certain things that I still consider to be unit testing
> > (like basic networking tests) that I don't want to have to write with
> > qtest.  I'm not up for writing a TCP/IP stack in Python...
>
> A ping test is not a unit test.
>
> The ping test is already covered by kvm-autotest; just set up a config
> that runs just that; after the initial run you'll have a guest installed
> so it'll be quick.  If we have a DSL guest it'll even be very quick.

Achieving this with kvm-autotest is not easy.  I doubt that DSL has an automated 
install mechanism and the step file based guest installation is fincky at best.

When we had the kvm-autotest day, none of the folks that tried to set it up who 
had never set it up before had something working by the end of the day.  And 
that was with lmr available on IRC to answer questions.

There is a huge setup cost with kvm-autotest.

> To test the various NIC emulations, you don't need a full TCP stack,
> just like you didn't need to write an NTP implementation for qtest's rtc
> test.  Instead you just poke values until it sends out a packet.  If you
> want to test virtio-net with both direct and indirect buffers, you can
> only do that with a unit test, you can't do it using a full linux guest
> since it has its own ideas of when to use indirects and when to avoid
> them (for example, it may choose to always avoid them).

I am not advocating that we don't write proper unit tests...

> > Acceptance testing is, "does Windows boot", "can I create three
> > virtio-serial devices".
> >
> > Obviously, part of acceptance testing is, "does this set of functional
> > tests pass".
>
> Seems like a very blurry line.  Especially as the functional test is
> weaker than either qtest and kvm-autotest.  I now have to agree with the
> others that it's duplicate functionality.  Does it really matter whether
> you're creating an image by compiling Linux and assembling and
> initramfs, or by downloading Fedora.iso and installing it?

Yes.  One of them is built entirely from source and the other is doing something 
meant to be done on bare metal.

How do you test ARM guests with kvm-autotest?  There is no such thing as an ISO 
installer for ARM.  Are we going to download random images from the internet?

Even if there is an ISO, installing a guest with TCG will take many hours.  My 
local qemu-test repo can now fully bootstrap an initramfs for non-x86 targets 
(which involves building gcc and uClibc).  Start to finish it takes about 30 
minutes regardless of the target (since the build runs natively).

> Would you add live migration testing to qemu-test?  If yes, you're
> duplicating some more.  If not, you're not doing functional or coverage
> tests for that functionality.

I would add live migration schema testing, absolutely.  I wouldn't add an 
acceptance test though.

An acceptance test with live migration would be creating a guest, running it, 
and then seeing it live migration worked.

> > I don't want to write a TCP/IP stack.  We aren't just grabbing a
> > random distro kernel.  We're building one from scratch configured in a
> > specific way.
>
> How does that help?

Not sure I understand the question.

Regards,

Anthony Liguori

> > In theory, we could even host the kernel source on git.qemu.org and
> > fork it to add more interesting things in the kernel (although I'd
> > prefer not to do this, obviously).
>
> That way lies madness, though it could be argued that it lies all around us.