Re: [PATCH v4 35/35] docs/devel/testing: Add documentation for functional tests

[Daniel P . Berrangé]

On Wed, Aug 21, 2024 at 10:27:36AM +0200, Thomas Huth wrote:
> Document the new functional testing framework. The text is originally
> based on the Avocado documentation, but heavily modified to match the
> new framework.
> 
> Signed-off-by: Thomas Huth <thuth@redhat.com>
> ---
>  docs/devel/testing/functional.rst | 269 ++++++++++++++++++++++++++++++
>  docs/devel/testing/index.rst      |   1 +
>  docs/devel/testing/main.rst       |  12 ++
>  3 files changed, 282 insertions(+)
>  create mode 100644 docs/devel/testing/functional.rst

Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>

> +The tests should be written in the style of the Python `unittest`_
> +framework, using stdio for the TAP protocol. The folder
> +``tests/functional/qemu_test`` provides classes (e.g. the ``QemuBaseTest``
> +and the ``QemuSystemTest`` classes) and utility functions that help
> +to get your test into the right shape.

One gotcha when using TAP protocol is that you can't just spew
debug info to stdout/stderr. Each line of debug info needs to
be prefixed with '#' so it is interpreted as diagnostic output.

We should point this out and recommend people to exclusively
using the 'logging' framework.

Particular care should be taken when spawning sub-processes
to capture both stdout/stderr and then log the result if
needed.


> +Overview
> +--------
> +
> +The ``tests/functional/qemu_test`` directory provides the ``qemu_test``
> +Python module, containing the ``qemu_test.QemuSystemTest`` class.
> +Here is a simple usage example:
> +
> +.. code::
> +
> +  #!/usr/bin/env python3
> +
> +  from qemu_test import QemuSystemTest
> +
> +  class Version(QemuSystemTest):
> +
> +      def test_qmp_human_info_version(self):
> +          self.vm.launch()
> +          res = self.vm.cmd('human-monitor-command',
> +                            command_line='info version')
> +          self.assertRegex(res, r'^(\d+\.\d+\.\d)')
> +
> +  if __name__ == '__main__':
> +      QemuSystemTest.main()
> +
> +By providing the "hash bang" line at the beginning of the script,
> +and by calling into QemuSystemTest.main() when it is run directly,
> +the test framework makes sure to run all test_*() functions in the
> +right fassion (e.g. with TAP output that is required by the meson test
> +runner).

Perhaps say that the test file should have execute permissions,
given the hash bang ?



With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|