Re: [Qemu-devel] [RFC][PATCH v6 00/04] qtest: qemu unit testing framework

[Stefan Hajnoczi]

On Wed, Feb 9, 2011 at 8:39 PM, Michael Roth <address@hidden> wrote:
> On 02/09/2011 01:42 PM, Blue Swirl wrote:
>>
>> On Fri, Feb 4, 2011 at 3:49 PM, Michael Roth<address@hidden>
>>  wrote:
>>>
>>> These patches apply to master (2-04-2011), and can also be obtained from:
>>> git://repo.or.cz/qemu/mdroth.git qtest_v1
>>>
>>> OVERVIEW:
>>>
>>> QEMU currently lacks a standard means to do targeted unit testing of the
>>> device model. Frameworks like kvm-autotest interact via guest OS, which
>>> provide a highly abstracted interface to the underlying machine, and are
>>> susceptable to bugs in the guest OS itself. This allows for reasonable test
>>> coverage of guest functionality as a whole, but reduces the accuracy and
>>> specificity with which we can exercise paths in the underlying devices.
>>>
>>> The following patches provide the basic beginnings of a test framework
>>> which replaces vcpu threads with test threads that interact with the
>>> underlying machine directly, allowing for directed unit/performance testing
>>> of individual devices. Test modules are built directly into the qemu binary,
>>> and each module provides the following interfaces:
>>>
>>> init():
>>>  Called in place of qemu's normal machine initialization to setup up
>>> devices explicitly. A full machine can be created here by calling into the
>>> normal init path, as well as minimal machines with a select set of
>>> buses/devices/IRQ handlers.
>>>
>>> run():
>>>  Test logic that interacts with the now-created machine.
>>>
>>> cleanup():
>>>  Currently unused, but potentially allows for chaining multiple tests
>>> together. Currently we run one module, then exit.
>>>
>>> As mentioned these are very early starting points. We're mostly looking
>>> for input from the community on the basic approach and overall requirements
>>> for an acceptable framework. A basic RTC test module is provided as an
>>> example.
>>>
>>> BUILD/EXAMPLE USAGE:
>>>
>>>  $ ./configure --target-list=x86_64-softmmu --enable-qtest
>>> --enable-io-thread
>>>  $ make
>>>  $ ./x86_64-softmmu/qemu-system-x86_64 -test ?
>>>  Available test modules:
>>>  rtc
>>>  $ ./x86_64-softmmu/qemu-system-x86_64 -test rtc
>>>  ../qtest/qtest_rtc.c:test_drift():L94: hz: 2, duration_ms: 4999,
>>> exp_duration: 5000, drift ratio: 0.000200
>>>  ../qtest/qtest_rtc.c:test_drift():L111: hz: 1024, duration_ms: 4999,
>>> exp_duration: 5000, drift ratio: 0.000200
>>>
>>> GENERAL PLAN:
>>>
>>>  - Provide libraries for common operations like PCI device enumeration,
>>> APIC configuration, default-configured machine setup, interrupt handling,
>>> etc.
>>>  - Develop tests as machine/target specific, potentially make some tests
>>> re-usable as interfaces are better defined
>>>  - Do port i/o via cpu_in/cpu_out commands
>>>  - Do guest memory access via a CPUPhysMemoryClient interface
>>>  - Allow interrupts to be sent by writing to an FD, detection in test
>>> modules via select()/read()
>>>
>>> TODO:
>>>
>>>  - A means to propagate test returns values to main i/o thread
>>>  - Better defined test harness for individual test cases and/or modules,
>>> likely via GLib
>>>  - Support for multiple test threads in a single test module for
>>> scalability testing
>>>  - Modify vl.c hooks so tests can configure their own timers/clocksources
>>>  - More test modules, improve current rtc module
>>>  - Further implementing/fleshing out of the overall plan
>>>
>>> Comments/feedback are welcome!
>>
>> Would it be possible to couple this with the tracing or Kemari somehow
>> so that you could capture, say, block device traces and feed them to
>> test setup?
>
> I would think so...it's a pretty open ended framework, a unit test could,
> say, read in block device traces in some pre-defined format and then execute
> those against a block device. We're also planning on adding command-line
> parameters for tests, so a unit test could actually be used as a general
> testing utility. for instance:

That's a good point.  Testing network, block, serial, etc device
emulation requires mock host devices (netdev, drive, chardev).

Net needs a "replay" net client.  A "dump" net client already exists
for capturing packets.

Block has no record/replay but the Linux blktrace format might be
good.  There's also Kevin's blkdebug and CQ's blksim which might be
extendable.

For chardev perhaps the existing options are already powerful enough,
otherwise something like expect would be neat.

> qemu -test block-trace-virtio -test-opts
> tracefile=<file>,target_img=<img>,target_fmt=qcow2,comparison_img=<img>

Or more like the -device, -chardev, etc syntax:

qemu -test 
block-trace-virtio,tracefile=<file>,target_img=<img>,target_fmt=qcow2,comparison_img=<img>

Stefan