qemu-devel
[Top][All Lists]
Advanced

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Qemu-devel] [PATCH] docs: Add docs/devel/testing.rst


From: Fam Zheng
Subject: Re: [Qemu-devel] [PATCH] docs: Add docs/devel/testing.rst
Date: Wed, 31 Jan 2018 11:15:35 +0800

On Mon, Jan 29, 2018 at 10:51 PM, Eric Blake <address@hidden> wrote:
> On 01/28/2018 09:31 PM, Fam Zheng wrote:
>> To make our efforts on QEMU testing easier to consume by contributors,
>> let's add a document. For example, Patchew reports build errors on
>> patches that should be relativly easy to reproduce with a few steps, and
>
> s/relativly/relatively/
>
>> it is much nicer if there is such a documentation that it can refer to.
>>
>> This focues on how to run existing tests and how to write new test
>
> s/focues/focuses/
>
>> cases, without going into the frameworks themselves.
>>
>> Signed-off-by: Fam Zheng <address@hidden>
>>
>> ---
>>
>
> I'll try not to repeat comments made by others...
>
>> +This document describes the testing infrastructure in QEMU.
>> +
>> +"Make check" testings
>> +=====================
>
> s/testings/tests/, or 'Testing with "make check"'

Sounds good, it will put "make" back to lower case again. :)

>
>> +
>> +The "make check" testing family includes most of the C based tests in QEMU. 
>> For
>> +a quick help, run ``make check-help`` from the source tree.
>> +
>> +The usual way to run these tests is:
>> +
>> +.. code::
>> +
>> +  make check
>> +
>> +which includes QAPI schema tests, unit tests, and QTests. Different 
>> sub-types
>> +of "make check" testings will be explained below.
>
> s/testings/tests/
>
>> +
>> +Before running tests, it is best to build QEMU programs first. Some tests
>> +expect the executables to exist and will fail with obscure messages if 
>> cannot
>> +find them.
>
> Should we fix 'make check' to depend on 'make all', so that we don't
> have to require this?

Maybe. We should try.

>
>> +
>> +Unit tests
>> +----------
>> +
>> +Unit tests, which can be invoked with ``make check-unit``, are simple C 
>> tests
>> +that typically link to individual QEMU objects and exercise them by calling
>> +into the modules.
>> +
>> +If you are writing new code in QEMU, consider adding a unit test, especially
>> +for utility modules that are relatively stateless or have few dependencies. 
>> To
>> +add a new unit test:
>> +
>> +1. Create a new source file. For example, ``tests/test-foo.c``.
>> +
>> +2. Write the test. Normally you would include the headers file which exports
>> +   the module API, then verify the interface behaves as expected from your
>> +   test. The test code should be organized with the glib testing framework.
>> +   Copy and modify an existing test is usually a good idea.
>> +
>> +3. Add the test to ``tests/Makefile.include``. First, name the unit test
>> +   program and add it to ``$(check-unit-y)``; then add a rule to build the
>> +   executable. Optionally, you can add a magical variable to support 
>> ``gcov``.
>> +   For example:
>> +
>> +.. code::
>> +
>> +  check-unit-y += tests/test-foo$(EXESUF)
>> +  tests/test-foo$(EXESUF): tests/test-foo.o $(test-util-obj-y)
>> +  ...
>> +  gcov-files-test-foo-y = util/foo.c
>
> Is it worth documenting that you can often run 'gdb ./test-foo' after
> the fact for a failing test?  (Most unit tests don't require any magic
> environment variables to be set)

Good point!

I'll also add a paragraph about MALLOC_PERTURB_ then.

>
>> +
>> +QTest
>> +-----
>> +
>> +QTest is a testing framework that simplifies starting QEMU and interacting 
>> with
>> +the virtual machine just like a guest kernel does. It can be very useful to
>> +test hardware emulation, for example; it could also control certain aspects 
>> of
>> +QEMU (such as virtual clock stepping), with a specially purposed "qtest"
>> +protocol. Refer to the documentation in ``qtest.c`` file for more details of
>> +the protocol.
>> +
>> +QTest cases can be executed with
>> +
>> +.. code::
>> +
>> +   make check-qtest
>> +
>> +The QTest library is implemented by ``tests/libqtest.c`` and the API is 
>> defined
>> +in ``tests/libqtest.h``.
>> +
>> +Consider adding a new QTest case when you are introducing a new virtual
>> +hardware, or extending one if you are adding functionalities to an existing
>> +virtual device.
>> +
>> +On top of libqtest, a higher level library, ``libqos``, was created to
>> +encapsulate common tasks of device drivers, such as memory management and
>> +communicating with system buses or devices. Many virtual device tests use
>> +libqos instead of directly calling into libqos.
>> +
>> +Steps to add a new QTest case are:
>> +
>> +1. Create a new source file for the test. (More than one file can be added 
>> as
>> +   necessary.) For example, ``tests/test-foo-device.c``.  2. Write the test
>> +   code with the glib and libqtest/libqos API. See also existing tests and 
>> the
>> +   library headers for reference.
>> +
>> +3. Register the new test in ``tests/Makefile.include``. Add the test 
>> executable
>> +   name to an appropriate ``check-qtest-*-y`` variable. For example:
>> +
>> +   ``check-qtest-generic-y = tests/test-foo-device$(EXESUF)``
>> +
>> +4. Add object dependencies of the executable in the Makefile, including the
>> +   test source file(s) and other interesting objects. For example:
>> +
>> +   ``tests/test-foo-device$(EXESUF): tests/test-foo-device.o 
>> $(libqos-obj-y)``
>
> Is it worth documenting that 'make check-qtest' runs EVERY test unless
> you define magic environment variables; and if it fails, running 'make
> check-qtest V=1' can help identify which test failed?  Here, running a
> mere 'gdb foo-device-test' is prone to fail (because it is missing magic
> environment variables); it's possible to run individual tests under gdb
> but it requires more effort (among other things, the V=1 helps you learn
> what environment variables to set).

Yes, will add a paragraph here too.

>
>> +
>
>> +Docker based testing
>> +====================
>
> 'Docker based tests' or 'Testing with docker'
>
>> +
>> +Introduction
>> +------------
>> +
>> +Docker testing framework in QEMU utilizes the public Docker images to build 
>> and
>
> s/^Docker/The Docker/
> s/the public/public/
>
>> +test QEMU in predefined and widely accessible Linux environments.  This 
>> makes
>> +it possible to expand the test coverage across distros, toolchain flavors 
>> and
>> +library versions.
>> +
>
>
> --
> Eric Blake, Principal Software Engineer
> Red Hat, Inc.           +1-919-301-3266
> Virtualization:  qemu.org | libvirt.org
>



reply via email to

[Prev in Thread] Current Thread [Next in Thread]