Re: [Qemu-devel] [PATCH v2] docs: Add docs/devel/testing.rst

[Andrew Jones]

On Wed, Jan 31, 2018 at 11:28:00AM +0800, Fam Zheng wrote:
> To make our efforts on QEMU testing easier to consume by contributors,
> let's add a document. For example, Patchew reports build errors on
> patches that should be relatively easy to reproduce with a few steps, and
> it is much nicer if there is such a documentation that it can refer to.
> 
> This focuses on how to run existing tests and how to write new test
> cases, without going into the frameworks themselves.
> 
> The VM based testing section is moved from tests/vm/README which now
> is a single line pointing to the new doc.
> 
> Signed-off-by: Fam Zheng <address@hidden>
> 
> ---
> 
> v2: Fix spelling errors and improve wordings. [Stefan, Eric, Thomas]
>     Example test name "test-foo -> foo-test". The mass renaming will be
>     done in another series. [Thomas, Eric]
>     Document how to debug unit tests and qtests. [Eric]
>     Document group permission setup for docker, and the privilege risks.
>     [Alex]
>     Update tests/vm/README.
> ---
>  docs/devel/testing.rst | 486 
> +++++++++++++++++++++++++++++++++++++++++++++++++
>  tests/vm/README        |  90 +--------
>  2 files changed, 487 insertions(+), 89 deletions(-)
>  create mode 100644 docs/devel/testing.rst
> 
> diff --git a/docs/devel/testing.rst b/docs/devel/testing.rst
> new file mode 100644
> index 0000000000..795df60dcd
> --- /dev/null
> +++ b/docs/devel/testing.rst
> @@ -0,0 +1,486 @@
> +===============
> +Testing in QEMU
> +===============
> +
> +This document describes the testing infrastructure in QEMU.
> +
> +Testing with "make check"
> +=========================
> +
> +The "make check" testing family includes most of the C based tests in QEMU. 
> For
> +a quick help, run ``make check-help`` from the source tree.
> +
> +The usual way to run these tests is:
> +
> +.. code::
> +
> +  make check
> +
> +which includes QAPI schema tests, unit tests, and QTests. Different sub-types
> +of "make check" testings will be explained below.
> +
> +Before running tests, it is best to build QEMU programs first. Some tests
> +expect the executables to exist and will fail with obscure messages if they
> +cannot find them.
> +
> +Unit tests
> +----------
> +
> +Unit tests, which can be invoked with ``make check-unit``, are simple C tests
> +that typically link to individual QEMU object files and exercise them by
> +calling exported functions.
> +
> +If you are writing new code in QEMU, consider adding a unit test, especially
> +for utility modules that are relatively stateless or have few dependencies. 
> To
> +add a new unit test:
> +
> +1. Create a new source file. For example, ``tests/foo-test.c``.
> +
> +2. Write the test. Normally you would include the header file which exports
> +   the module API, then verify the interface behaves as expected from your
> +   test. The test code should be organized with the glib testing framework.
> +   Copying and modifying an existing test is usually a good idea.
> +
> +3. Add the test to ``tests/Makefile.include``. First, name the unit test
> +   program and add it to ``$(check-unit-y)``; then add a rule to build the
> +   executable. Optionally, you can add a magical variable to support 
> ``gcov``.
> +   For example:
> +
> +.. code::
> +
> +  check-unit-y += tests/foo-test$(EXESUF)
> +  tests/foo-test$(EXESUF): tests/foo-test.o $(test-util-obj-y)
> +  ...
> +  gcov-files-foo-test-y = util/foo.c
> +
> +Since unit tests don't require environment variables, the simplest way to 
> debug
> +a unit test failure is often directly invoking it or even running it under
> +``gdb``. However there can still be differences in behavior between ``make``
> +invocations and your manual run, due to ``$MALLOC_PERTURB_`` environment
> +variable (which affects memory reclaimation and catches invalid pointers
> +better) and gtester options. If necessary, you can run
> +
> +.. code::
> +  make check-unit V=1
> +
> +and copy the actual command line which executes the unit test, then run
> +it from the command line.
> +
> +QTest
> +-----
> +
> +QTest is a device emulation testing framework.  It can be very useful to test
> +device models; it could also control certain aspects of QEMU (such as virtual
> +clock stepping), with a special purpose "qtest" protocol.  Refer to the
> +documentation in ``qtest.c`` for more details of the protocol.
> +
> +QTest cases can be executed with
> +
> +.. code::
> +
> +   make check-qtest
> +
> +The QTest library is implemented by ``tests/libqtest.c`` and the API is 
> defined
> +in ``tests/libqtest.h``.
> +
> +Consider adding a new QTest case when you are introducing a new virtual
> +hardware, or extending one if you are adding functionalities to an existing
> +virtual device.
> +
> +On top of libqtest, a higher level library, ``libqos``, was created to
> +encapsulate common tasks of device drivers, such as memory management and
> +communicating with system buses or devices. Many virtual device tests use
> +libqos instead of directly calling into libqos.
                                           ^^ libqtest?

Thanks,
drew