[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [Qemu-devel] [PATCH 2/2] Acceptance tests: add make rule for running
From: |
Eduardo Habkost |
Subject: |
Re: [Qemu-devel] [PATCH 2/2] Acceptance tests: add make rule for running them |
Date: |
Thu, 20 Sep 2018 15:58:57 -0300 |
User-agent: |
Mutt/1.9.2 (2017-12-15) |
On Thu, Sep 20, 2018 at 11:19:56AM -0400, Cleber Rosa wrote:
> The acceptance (aka functional, aka Avocado-based) tests are
> Python files located in "tests/acceptance" that need to be run
> with the Avocado libs and test runner.
>
> Let's provide a convenient way for QEMU developers to run them,
> by making use of the tests-venv with the required setup.
>
> Also, while the Avocado test runner will take care of creating a
> location to save test results to, it was understood that it's better
> if the results are kept within the build tree.
>
> Signed-off-by: Cleber Rosa <address@hidden>
> ---
> docs/devel/testing.rst | 28 +++++++++++++++++++++++-----
> tests/Makefile.include | 17 +++++++++++++++--
> tests/venv-requirements.txt | 1 +
> 3 files changed, 39 insertions(+), 7 deletions(-)
>
> diff --git a/docs/devel/testing.rst b/docs/devel/testing.rst
> index 727c4019b5..0fbf0d0aac 100644
> --- a/docs/devel/testing.rst
> +++ b/docs/devel/testing.rst
> @@ -545,10 +545,24 @@ Tests based on ``avocado_qemu.Test`` can easily:
> -
> http://avocado-framework.readthedocs.io/en/latest/api/test/avocado.html#avocado.Test
> -
> http://avocado-framework.readthedocs.io/en/latest/api/utils/avocado.utils.html
>
> -Installation
> -------------
> +Running tests
> +-------------
>
> -To install Avocado and its dependencies, run:
> +You can run the acceptance tests simply by executing:
> +
> +.. code::
> +
> + make check-acceptance
> +
> +This involves the automatic creation of Python virtual environment
> +within the build tree (at ``tests/venv``) which will have all the
> +right dependencies, and will save tests results also within the
> +build tree (at ``tests/results``).
> +
> +Manual Installation
> +-------------------
> +
> +To manually install Avocado and its dependencies, run:
>
> .. code::
>
> @@ -689,11 +703,15 @@ The exact QEMU binary to be used on QEMUMachine.
> Uninstalling Avocado
> --------------------
>
> -If you've followed the installation instructions above, you can easily
> -uninstall Avocado. Start by listing the packages you have installed::
> +If you've followed the manual installation instructions above, you can
> +easily uninstall Avocado. Start by listing the packages you have
> +installed::
>
> pip list --user
>
> And remove any package you want with::
>
> pip uninstall <package_name>
> +
> +If you've used ``make check-acceptance``, the Python virtual environment
> where
> +Avocado is installed will be cleaned up as part of ``make check-clean``.
> diff --git a/tests/Makefile.include b/tests/Makefile.include
> index 9bb90a83d4..8cef694954 100644
> --- a/tests/Makefile.include
> +++ b/tests/Makefile.include
> @@ -11,6 +11,7 @@ check-help:
> @echo " $(MAKE) check-qapi-schema Run QAPI schema tests"
> @echo " $(MAKE) check-block Run block tests"
> @echo " $(MAKE) check-tcg Run TCG tests"
> + @echo " $(MAKE) check-acceptance Run all acceptance (functional)
> tests"
> @echo " $(MAKE) check-report.html Generates an HTML test report"
> @echo " $(MAKE) check-venv Creates a Python venv for tests"
> @echo " $(MAKE) check-clean Clean the tests"
> @@ -1002,10 +1003,11 @@ check-decodetree:
>
> # Python venv for running tests
>
> -.PHONY: check-venv
> +.PHONY: check-venv check-acceptance
>
> TESTS_VENV_DIR=$(BUILD_DIR)/tests/venv
> TESTS_VENV_REQ=$(BUILD_DIR)/tests/venv-requirements.txt
> +TESTS_RESULTS_DIR=$(BUILD_DIR)/tests/results
>
> $(TESTS_VENV_DIR):
> $(call quiet-command, \
> @@ -1015,8 +1017,19 @@ $(TESTS_VENV_DIR):
> $(TESTS_VENV_DIR)/bin/pip -q install -r $(TESTS_VENV_REQ), \
> PIP, $(TESTS_VENV_REQ))
>
> +$(TESTS_RESULTS_DIR):
> + $(call quiet-command, mkdir -p $@, \
> + MKDIR, $@)
> +
> check-venv: $(TESTS_VENV_DIR)
>
> +check-acceptance: check-venv $(TESTS_RESULTS_DIR)
> + $(call quiet-command, \
> + $(TESTS_VENV_DIR)/bin/avocado \
> + --show=none run --job-results-dir=$(TESTS_RESULTS_DIR)
> --failfast=on \
> + $(SRC_PATH)/tests/acceptance, \
> + "AVOCADO", "tests/acceptance")
I think we should provide something easy to use for people who
already have the right Avocado version installed in their system
and want to avoid re-downloading Avocado every time.
We already have plans to do this automatically/transparently in
the future, but maybe while we don't have something automatic we
could have two separate rules. e.g.:
AVOCADO = avocado
check-acceptance: $(TESTS_RESULTS_DIR)
$(call quiet-command, \
$(AVOCADO) \
--show=none run --job-results-dir=$(TESTS_RESULTS_DIR)
--failfast=on \
$(SRC_PATH)/tests/acceptance, \
"AVOCADO", "tests/acceptance")
check-acceptance-venv: check-venv
$(MAKE) check-acceptance AVOCADO=$(TESTS_VENV_DIR)/bin/avocado
> +
> # Consolidated targets
>
> .PHONY: check-qapi-schema check-qtest check-unit check check-clean
> @@ -1030,7 +1043,7 @@ check-clean:
> rm -rf $(check-unit-y) tests/*.o $(QEMU_IOTESTS_HELPERS-y)
> rm -rf $(sort $(foreach target,$(SYSEMU_TARGET_LIST),
> $(check-qtest-$(target)-y)) $(check-qtest-generic-y))
> rm -f tests/test-qapi-gen-timestamp
> - rm -rf $(TESTS_VENV_DIR)
> + rm -rf $(TESTS_VENV_DIR) $(TESTS_RESULTS_DIR)
>
> clean: check-clean
>
> diff --git a/tests/venv-requirements.txt b/tests/venv-requirements.txt
> index d39f9d1576..1734d0ce27 100644
> --- a/tests/venv-requirements.txt
> +++ b/tests/venv-requirements.txt
> @@ -1,3 +1,4 @@
> # Add Python module requirements, one per line, to be installed
> # in the tests/venv Python virtual environment. For more info,
> # refer to: https://pip.pypa.io/en/stable/user_guide/#id1
> +avocado-framework==64.0
> --
> 2.17.1
>
--
Eduardo
- [Qemu-devel] [PATCH] Bootstrap Python venv and acceptance/functional tests, Cleber Rosa, 2018/09/20
- [Qemu-devel] [PATCH 2/2] Acceptance tests: add make rule for running them, Cleber Rosa, 2018/09/20
- Re: [Qemu-devel] [PATCH 2/2] Acceptance tests: add make rule for running them,
Eduardo Habkost <=
- Re: [Qemu-devel] [PATCH 2/2] Acceptance tests: add make rule for running them, Cleber Rosa, 2018/09/20
- Re: [Qemu-devel] [PATCH 2/2] Acceptance tests: add make rule for running them, Eduardo Habkost, 2018/09/20
- Re: [Qemu-devel] [PATCH 2/2] Acceptance tests: add make rule for running them, Cleber Rosa, 2018/09/20
- Re: [Qemu-devel] [PATCH 2/2] Acceptance tests: add make rule for running them, Philippe Mathieu-Daudé, 2018/09/20
- Re: [Qemu-devel] [PATCH 2/2] Acceptance tests: add make rule for running them, Eduardo Habkost, 2018/09/20
- Re: [Qemu-devel] [PATCH 2/2] Acceptance tests: add make rule for running them, Philippe Mathieu-Daudé, 2018/09/20
- Re: [Qemu-devel] [PATCH 2/2] Acceptance tests: add make rule for running them, Cleber Rosa, 2018/09/21
- Re: [Qemu-devel] [PATCH 2/2] Acceptance tests: add make rule for running them, Cleber Rosa, 2018/09/21
- Re: [Qemu-devel] [PATCH 2/2] Acceptance tests: add make rule for running them, Markus Armbruster, 2018/09/21
- Re: [Qemu-devel] [PATCH 2/2] Acceptance tests: add make rule for running them, Eduardo Habkost, 2018/09/21