[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [Qemu-devel] [PATCH 2/2] Acceptance tests: add make rule for running
From: |
Markus Armbruster |
Subject: |
Re: [Qemu-devel] [PATCH 2/2] Acceptance tests: add make rule for running them |
Date: |
Fri, 21 Sep 2018 15:34:02 +0200 |
User-agent: |
Gnus/5.13 (Gnus v5.13) Emacs/26.1 (gnu/linux) |
Eduardo Habkost <address@hidden> writes:
> On Fri, Sep 21, 2018 at 08:36:22AM +0200, Markus Armbruster wrote:
>> Eduardo Habkost <address@hidden> writes:
>>
>> > On Thu, Sep 20, 2018 at 11:19:56AM -0400, Cleber Rosa wrote:
>> >> The acceptance (aka functional, aka Avocado-based) tests are
>> >> Python files located in "tests/acceptance" that need to be run
>> >> with the Avocado libs and test runner.
>> >>
>> >> Let's provide a convenient way for QEMU developers to run them,
>> >> by making use of the tests-venv with the required setup.
>> >>
>> >> Also, while the Avocado test runner will take care of creating a
>> >> location to save test results to, it was understood that it's better
>> >> if the results are kept within the build tree.
>> >>
>> >> Signed-off-by: Cleber Rosa <address@hidden>
>> >> ---
>> [...]
>> >> diff --git a/tests/Makefile.include b/tests/Makefile.include
>> >> index 9bb90a83d4..8cef694954 100644
>> >> --- a/tests/Makefile.include
>> >> +++ b/tests/Makefile.include
>> >> @@ -11,6 +11,7 @@ check-help:
>> >> @echo " $(MAKE) check-qapi-schema Run QAPI schema tests"
>> >> @echo " $(MAKE) check-block Run block tests"
>> >> @echo " $(MAKE) check-tcg Run TCG tests"
>> >> + @echo " $(MAKE) check-acceptance Run all acceptance (functional)
>> >> tests"
>> >> @echo " $(MAKE) check-report.html Generates an HTML test report"
>> >> @echo " $(MAKE) check-venv Creates a Python venv for tests"
>> >> @echo " $(MAKE) check-clean Clean the tests"
>> >> @@ -1002,10 +1003,11 @@ check-decodetree:
>> >>
>> >> # Python venv for running tests
>> >>
>> >> -.PHONY: check-venv
>> >> +.PHONY: check-venv check-acceptance
>> >>
>> >> TESTS_VENV_DIR=$(BUILD_DIR)/tests/venv
>> >> TESTS_VENV_REQ=$(BUILD_DIR)/tests/venv-requirements.txt
>> >> +TESTS_RESULTS_DIR=$(BUILD_DIR)/tests/results
>> >>
>> >> $(TESTS_VENV_DIR):
>> >> $(call quiet-command, \
>> >> @@ -1015,8 +1017,19 @@ $(TESTS_VENV_DIR):
>> >> $(TESTS_VENV_DIR)/bin/pip -q install -r $(TESTS_VENV_REQ), \
>> >> PIP, $(TESTS_VENV_REQ))
>> >>
>> >> +$(TESTS_RESULTS_DIR):
>> >> + $(call quiet-command, mkdir -p $@, \
>> >> + MKDIR, $@)
>> >> +
>> >> check-venv: $(TESTS_VENV_DIR)
>> >>
>> >> +check-acceptance: check-venv $(TESTS_RESULTS_DIR)
>> >> + $(call quiet-command, \
>> >> + $(TESTS_VENV_DIR)/bin/avocado \
>> >> + --show=none run --job-results-dir=$(TESTS_RESULTS_DIR)
>> >> --failfast=on \
>> >> + $(SRC_PATH)/tests/acceptance, \
>> >> + "AVOCADO", "tests/acceptance")
>> >
>> > I think we should provide something easy to use for people who
>> > already have the right Avocado version installed in their system
>> > and want to avoid re-downloading Avocado every time.
>> >
>> > We already have plans to do this automatically/transparently in
>> > the future, but maybe while we don't have something automatic we
>> > could have two separate rules. e.g.:
>> >
>> > AVOCADO = avocado
>> >
>> > check-acceptance: $(TESTS_RESULTS_DIR)
>> > $(call quiet-command, \
>> > $(AVOCADO) \
>> > --show=none run --job-results-dir=$(TESTS_RESULTS_DIR)
>> > --failfast=on \
>> > $(SRC_PATH)/tests/acceptance, \
>> > "AVOCADO", "tests/acceptance")
>> >
>> > check-acceptance-venv: check-venv
>> > $(MAKE) check-acceptance AVOCADO=$(TESTS_VENV_DIR)/bin/avocado
>>
>> Recursion just to override a variable feels like overkill.
>> Have you considered target-specific variable values?
>>
>> https://www.gnu.org/software/make/manual/html_node/Target_002dspecific.html#Target_002dspecific
>
> I have, but I wasn't sure what would be the pitfalls when doing
> that.
>
> For example, the following seems to work but makes the build
> results depend on the ordering of targets:
>
> $ cat Makefile
> VAR=x
> a:
> echo $(VAR)
> b: VAR=b
> b: a
> $ make a
> echo x
> x
> $ make b
> echo b
> b
> $ make a b
> echo x
> x
> make: Nothing to be done for 'b'.
> $ make b a
> echo b
> b
> make: 'a' is up to date.
> $
I doubt that's a problem for practical applications. Why would anyone
ask for two conflicting check-acceptance runs, and expect to get *both*?
- Re: [Qemu-devel] [PATCH 2/2] Acceptance tests: add make rule for running them, (continued)
- Re: [Qemu-devel] [PATCH 2/2] Acceptance tests: add make rule for running them, Cleber Rosa, 2018/09/20
- Re: [Qemu-devel] [PATCH 2/2] Acceptance tests: add make rule for running them, Eduardo Habkost, 2018/09/20
- Re: [Qemu-devel] [PATCH 2/2] Acceptance tests: add make rule for running them, Cleber Rosa, 2018/09/20
- Re: [Qemu-devel] [PATCH 2/2] Acceptance tests: add make rule for running them, Philippe Mathieu-Daudé, 2018/09/20
- Re: [Qemu-devel] [PATCH 2/2] Acceptance tests: add make rule for running them, Eduardo Habkost, 2018/09/20
- Re: [Qemu-devel] [PATCH 2/2] Acceptance tests: add make rule for running them, Philippe Mathieu-Daudé, 2018/09/20
- Re: [Qemu-devel] [PATCH 2/2] Acceptance tests: add make rule for running them, Cleber Rosa, 2018/09/21
- Re: [Qemu-devel] [PATCH 2/2] Acceptance tests: add make rule for running them, Cleber Rosa, 2018/09/21
- Re: [Qemu-devel] [PATCH 2/2] Acceptance tests: add make rule for running them, Markus Armbruster, 2018/09/21
- Re: [Qemu-devel] [PATCH 2/2] Acceptance tests: add make rule for running them, Eduardo Habkost, 2018/09/21
- Re: [Qemu-devel] [PATCH 2/2] Acceptance tests: add make rule for running them,
Markus Armbruster <=
- Re: [Qemu-devel] [PATCH 2/2] Acceptance tests: add make rule for running them, Eduardo Habkost, 2018/09/21
- Re: [Qemu-devel] [PATCH 2/2] Acceptance tests: add make rule for running them, Cleber Rosa, 2018/09/21
[Qemu-devel] [PATCH 1/2] Bootstrap Python venv for tests, Cleber Rosa, 2018/09/20