Re: [Qemu-devel] [PATCH 0/4] add "make check"

[Eduardo Habkost]

On Tue, Oct 25, 2011 at 03:27:35PM +0200, Gerd Hoffmann wrote:
[...]
> > A while ago I played with some simple IDE tests. It basically was a
> > small x86 kernel with an empty image that sends IDE commands and prints
> > some results, and a script that invokes the guest and checks whether the
> > test has passed or failed.
> 
> That reminds me that I've started toying with running tests inside a
> guest too.  Stopped working on it a while back due to other priorities.
>  Attached what I have so far.
> 
> > So at first I started with my own multiboot kernel and copied over some
> > parts of kvm-unittest's libc. Clearly not the best idea once it's more
> > than a couple of lines, so at some point I took the code and integrated
> > with my real kvm-unittests repository.
> > 
> > Now I don't have to duplicate code any more, but at the same time
> > there's no chance that a 'make check' in an upstream qemu tree could run
> > this. Tests for other devices will have exactly the same problem.
> > 
> > Any suggestions on how to go forward with this kind of tests? Should
> > this go into qemu or into kvm-unittests? Or should kvm-unittests be
> > merged into the qemu tree? Or is the approach completely wrong?
> 
> I think we should have some framework to run tests inside the guest in
> the qemu source tree.  I'm not sure kvm-unittests is the right tool for
> the job though.  It is quite low-level and mainly targets the kvm bits
> inside the linux kernel.  Testing -- for example -- usb device emulation
> would pretty much require writing a usb stack for kvm-unitests ...

We have a framework to run tests inside a fully-installed guest, that's
KVM-Autotest. But maybe it's too much for the kind of tests you need, I
don't know. There are different "levels" of testing, with different
reequirements, and we need to have good tools for all levels.

Just trying to enumerate the kind of tests somebody may need:

A) Simple unit tests for internal qemu C functions
   - 'make check' can run them, using either libcheck or gtest.
B) Functional tests that tests actual virtualization/emulation, but only
   of some specific subsystems, not using a fully-featured qemu process.
   - We don't have anything like that, today, right? I am not sure we
     need it.
C) Functional tests that just need to run a small binary with no OS
   installed in the guest, but running a fully-feature qemu process.
   - The tests in the 'tests' directory do this, right? kvm-unittests
     does this, right?
D) Functional tests that need a minimal OS installed, with, e.g., at
   least a Linux kernel and a shell.
   - This is what Gerd's patch below does, right? Also, KVM-Autotest can
     be used for this.
E) Functional tests that need a full OS installed and configured.
   - Today we use KVM-Autotest for this.


Does the above model look correct/complete, or is there some case I
missed?


> 
> cheers,
>   Gerd

> From 096f68ea08c3c4baf1bbdc549b257a67ecc87e25 Mon Sep 17 00:00:00 2001
> From: Gerd Hoffmann <address@hidden>
> Date: Tue, 13 Sep 2011 17:38:37 +0200
> Subject: [PATCH] initramfs test framework
> 
> Signed-off-by: Gerd Hoffmann <address@hidden>
> ---
>  initramfs/.gitignore         |    3 +
>  initramfs/10-qemu-udev.rules |    5 +
>  initramfs/Makefile           |   36 +++++++
>  initramfs/README             |   44 ++++++++
>  initramfs/init.c             |  225 
> ++++++++++++++++++++++++++++++++++++++++++
>  initramfs/initramfs-boot     |   32 ++++++
>  initramfs/initramfs-create   |  111 +++++++++++++++++++++
>  initramfs/test-ehci          |    3 +
>  initramfs/test-ehci.good     |    8 ++
>  initramfs/test-hello.c       |    7 ++
>  initramfs/test-hello.good    |    1 +
>  initramfs/test-uhci          |    3 +
>  initramfs/test-uhci.good     |    3 +
>  13 files changed, 481 insertions(+), 0 deletions(-)
>  create mode 100644 initramfs/.gitignore
>  create mode 100644 initramfs/10-qemu-udev.rules
>  create mode 100644 initramfs/Makefile
>  create mode 100644 initramfs/README
>  create mode 100644 initramfs/init.c
>  create mode 100755 initramfs/initramfs-boot
>  create mode 100755 initramfs/initramfs-create
>  create mode 100755 initramfs/test-ehci
>  create mode 100644 initramfs/test-ehci.good
>  create mode 100644 initramfs/test-hello.c
>  create mode 100644 initramfs/test-hello.good
>  create mode 100755 initramfs/test-uhci
>  create mode 100644 initramfs/test-uhci.good
> 

-- 
Eduardo