qemu-devel
[Top][All Lists]
Advanced

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [RFC PATCH 4/8] tests/pytest: add pytest to the meson build system


From: Thomas Huth
Subject: Re: [RFC PATCH 4/8] tests/pytest: add pytest to the meson build system
Date: Fri, 12 Jul 2024 12:14:45 +0200
User-agent: Mozilla Thunderbird

On 12/07/2024 11.01, Daniel P. Berrangé wrote:
On Thu, Jul 11, 2024 at 01:55:42PM +0200, Thomas Huth wrote:
From: Ani Sinha <ani@anisinha.ca>

Integrate the pytest framework with the meson build system. This
will make meson run all the pytests under the pytest directory.

Lets add a note about the compelling benefit of this new approach

   With this change, each functional test becomes subject
   to an individual execution timeout, defaulting to 60
   seconds, but overridable per-test.

The avocado runner uses timeouts, too, so it's not really an additional benefit that we get here.

For CI purposes we'll need to add 'python3-pytest' to
tests/lcitool/projects/qemu.yml, and re-generate the
the dockerfiles. Some of the other non-gitlab CI
integrations probably need manual additions of pytest
packages.

I'm currently rather looking into getting rid of pytest and to use pycotap instead: Using the TAP protocol for running the tests, you get a much nicer output from the meson test runner, which can then count the subtests and properly report SKIPs for tests that have not been run.

diff --git a/tests/Makefile.include b/tests/Makefile.include
index d39d5dd6a4..68151717d7 100644
--- a/tests/Makefile.include
+++ b/tests/Makefile.include
@@ -3,12 +3,14 @@
  .PHONY: check-help
  check-help:
        @echo "Regression testing targets:"
-       @echo " $(MAKE) check                  Run block, qapi-schema, unit, 
softfloat, qtest and decodetree tests"
+       @echo " $(MAKE) check                  Run block, qapi-schema, unit, 
softfloat, qtest, pytest and decodetree tests"
        @echo " $(MAKE) bench                  Run speed tests"
        @echo
        @echo "Individual test suites:"
        @echo " $(MAKE) check-qtest-TARGET     Run qtest tests for given target"
        @echo " $(MAKE) check-qtest            Run qtest tests"
+       @echo " $(MAKE) check-pytest           Run pytest tests"
+       @echo " $(MAKE) check-pytest-TARGET    Run pytest for a given target"

Or name it after the type of test rather than harness ?

  eg  check-functional / check-functional-TARGET

For that matter perhaps also for the dir name ?

    tests/functional/*.py

I almost expected that discussion again ... (see https://lists.gnu.org/archive/html/qemu-devel/2021-05/msg06553.html ) ... last time we couldn't really agree on such a name and decided to go with the name of the framework...

I agree that "pytest" is likely not the best name here, especially if switching to the pycotap test runner instead of using the "pytest" program, but "functional" might trigger the same discussion again as last time ... should it rather be "functional" or "validation" or "integration" etc.?

Maybe best if we come up with a new fictional name for the "new" test framework... something like "pyqe" - PYthon-based Qemu test Environment"? ... could be considered as a play on the word "pike", too, i.e. something that makes sure that not everything gets in ... ? WDYT?

 Thomas




reply via email to

[Prev in Thread] Current Thread [Next in Thread]