Re: [Qemu-devel] [ANNOUNCE] qemu-test: a set of tests scripts for QEMU

[Dor Laor]

On 12/19/2011 07:13 PM, Anthony Liguori wrote:
> Hi,
>
> I've published a set of tests I wrote over the weekend on qemu.org. My
> motivations were 1) to prevent regressions like the libguestfs one and
> 2) to have an easier way to do development testing as I work on QEMU
> Object Model.
>
> Now before sending the obligatory, "What about using KVM autotest"
> reply, note that this is significantly different than KVM autotest and
> really occupies a different use-case.

Well, I'm still not convinced that a new standalone package should 
handle these cases instead of kvm autotest. I'll be happy to integrate 
the tests to kvm autotest anyway and the more the merrier but imho it's 
a duplicate.

kvm-unit-test is a case for unit tests since it replaces the guest 
kernel itself and test for really low level tests like task switch, 
instruction emulation, etc.

> It has the following characteristics:
>
> 1) It builds a custom kernel and initramfs based on busybox. This is
> fairly important to ensure that we can run tests with no device
> pre-requisites.

This can be done easily w/ autotest too.

> 2) Tests are scripts that are launched in the initramfs
>
> 3) The test controls exactly how QEMU is launched which allows easy
> testing of various QEMU options

afaik kvm autotest do the same. It's true that kvm autotest might look 
less friendly but its may only be this way since it has lots of options.

> 4) Tests can interact with the monitor via QMP while the test is running

ditto for kvm autotest

> 5) The tests execute very quickly, can be run stand alone, and do not
> require root privileges

ditto for kvm auotest. It's possible to configure it w/o root too which 
is not a huge issue.

Please compare your own virtio-serial test w/ the autotest version of 
it: 
https://github.com/autotest/autotest/blob/master/client/tests/kvm/tests/virtio_console.py

This single file tests functionality, limits, console, live migration 
and performance. Of course one can add a very basic 'hello world' sanity 
test too that will run quickly and will identify basic breakage fast.

Noways we abstract kvm autotest so libvirt will be optionally tested too 
w/ the same tests.

Again, I'm not advocating that kvm autotest is a solution for anything 
but for plain guest-host communication, monitor commands, etc it's a 
really good tool.

I agree that kvm autotest may be less friendly for developer users since 
it carries allot of options for testing a huge matrix. Lucas and Cleber 
are working these days to add a make kvmautotest target to qemu so one 
would be able to quickly execute autotest and we can think of additional 
parameters like sanity-set, migration-set, etc.

> 6) They are random by nature with the ability to fix the seed in order
> to be used in git-bisect.
>
> I think Gerd had been looking at doing something similar with a custom
> initrd.
>
> I've tried to consider other architectures and had hoped that we could
> commit the vmlinuz and initramfs into git so that it was easy to test
> other architectures without having a full build environment.
> Unfortunately, busybox doesn't link statically with glibc and I can't
> see an obvious way to commit binaries while respecting the GPL since we
> need to pull glibc into the initramfs.
>
> I know buildroot exists specifically to deal with this but in my
> experience, buildroot is very unreliable and extremely heavy weight
> since it rebuilds gcc multiple times in order to bootstrap a ulibc
> environment.
>
> Anyway, the code is available at:
>
> http://git.qemu.org/qemu-test.git
>
> See the README for instructions on how to use it.
>
> Regards,
>
> Anthony Liguori

> --- Begin Message ---
>
>
>
> Subject: 
>
> [Qemu-devel] [PATCH] qemu-test: add virtio-serial test
>
>
>
>
> Date: 
>
> Thu, 22 Dec 2011 08:23:25 -0600
>
>
>
>
> This is a pretty simple test that just confirms that virtio-serial shows up and
> is writable.  It also tests the alias for virtio-serial-pci.
>
> Signed-off-by: Anthony Liguori <address@hidden>
> ---
>  tests/virtio-serial.sh |   52 ++++++++++++++++++++++++++++++++++++++++++++++++
>  1 files changed, 52 insertions(+), 0 deletions(-)
>  create mode 100755 tests/virtio-serial.sh
>
> diff --git a/tests/virtio-serial.sh b/tests/virtio-serial.sh
> new file mode 100755
> index 0000000..e95ae6e
> --- /dev/null
> +++ b/tests/virtio-serial.sh
> @@ -0,0 +1,52 @@
> +#!/bin/sh
> +
> +canary="** waiting for... **"
> +
> +in_host() {
> +    tmpchr=$tmpdir/chr.log
> +
> +    # Also test alias
> +    devname=`choose virtio-serial virtio-serial-pci`
> +
> +    qemu -nographic -enable-kvm -device $devname \
> +        -device virtserialport,name=org.qemu.test,chardev=chr0 \
> +        -chardev file,path=$tmpchr,id=chr0
> +    rc=$?
> +
> +    if test $rc = 0; then
> +       if ! grep "$canary" $tmpchr >/dev/null; then
> +           echo "Failed to see output from guest!"
> +           rc=1
> +       fi
> +    fi
> +
> +    rm -f $tmpchr
> +
> +    return $rc
> +}
> +
> +in_guest() {
> +    sysfspath=/sys/bus/virtio/devices/virtio0/virtio-ports/vport0p1
> +    if ! test -e $sysfspath/name; then
> +       echo "Device not visible!"
> +       return 1
> +    fi
> +
> +    name=`cat $sysfspath/name`
> +
> +    if test "$name" != "org.qemu.test"; then
> +       echo "Device has wrong name!"
> +       echo "Expected 'org.qemu.test', got '$name'"
> +       return 2
> +    fi
> +
> +    echo "$canary" > /dev/vport0p1
> +
> +    return 0
> +}
> +
> +if test $QEMU_TEST; then
> +    in_host
> +else
> +    in_guest
> +fi
> -- 
> 1.7.4.1
>
>
>
> --- End Message ---