Re: [Qemu-devel] [PATCH v4 0/3] Bootstrap Python venv and acceptance/functional tests

[Philippe Mathieu-Daudé]

On 16/10/2018 16:27, Cleber Rosa wrote:
> On 10/12/18 5:44 PM, Philippe Mathieu-Daudé wrote:
>> On 12/10/2018 18:53, Cleber Rosa wrote:
>>> TL;DR
>>> =====
>>>
>>> Allow acceptance tests to be run with `make check-acceptance`.
>>>
>>> Details
>>> =======
>>>
>>> This introduces a Python virtual environment that will be setup within
>>> the QEMU build directory, that will contain the exact environment that
>>> tests may require.
>>>
>>> There's one current caveat: it requires Python 3, as it's based on the
>>> venv module.  This was based on some discussions and perception about
>>> standardizing on Python 3, but can easily be made to accommodate Python
>>> 2 as well.
>>>
>>> Example of bootstrap and test execution on Travis-CI:
>>>
>>> https://travis-ci.org/qemu/qemu/jobs/439331028#L2508
>>
>> If you activate Travis on your github account, you can test that in your
>> namespace without having to open zombie pull requests there... A simple
>> push to your repository will trigger a full Travis build.
>>
>> This is how I use it btw, canceling the jobs I'm not interested in, to
>> quickly run the others.
>> i.e. https://travis-ci.org/philmd/qemu/jobs/439573299#L5600
>>
> 
> Yep, I've done that shortly after sending this. :)  Thanks for the tip,
> though.
> 
> One thing I miss is the ability, without editing the .travis.yaml, to
> "choose" just one/few jobs in a build.  Or maybe even the order they run
> (Of course I want the acceptance tests job first).
> 
> Do you know anything about that?

I just add an extra commit that drop every tests but the ones I'm
interested to run and keep this patch until I'm happy with the series.
I push as 'myseries_only' then I simply do "git branch myseries_only~
myseries && git push series" to run against everything, go to check my
mailbox and review patches until it finishes, then I redact the cover
and post.

> 
> - Cleber.
> 
>>>
>>>    ...
>>>       VENV    /home/travis/build/qemu/qemu/tests/venv
>>>       MKDIR   /home/travis/build/qemu/qemu/tests/results
>>>       PIP     /home/travis/build/qemu/qemu/tests/venv-requirements.txt
>>>       AVOCADO tests/acceptance
>>>     JOB ID     : 920e4fcf55a1782f1ae77bee64b20ccdc2e1111d
>>>     JOB LOG    : 
>>> /home/travis/build/qemu/qemu/tests/results/job-2018-10-09T21.42-920e4fc/job.log
>>>      (1/6) 
>>> /home/travis/build/qemu/qemu/tests/acceptance/boot_linux_console.py:BootLinuxConsole.test:
>>>   PASS (3.57 s)
>>>      (2/6) 
>>> /home/travis/build/qemu/qemu/tests/acceptance/version.py:Version.test_qmp_human_info_version:
>>>   PASS (0.04 s)
>>>      (3/6) 
>>> /home/travis/build/qemu/qemu/tests/acceptance/vnc.py:Vnc.test_no_vnc:  PASS 
>>> (0.04 s)
>>>      (4/6) 
>>> /home/travis/build/qemu/qemu/tests/acceptance/vnc.py:Vnc.test_no_vnc_change_password:
>>>   PASS (0.04 s)
>>>      (5/6) 
>>> /home/travis/build/qemu/qemu/tests/acceptance/vnc.py:Vnc.test_vnc_change_password_requires_a_password:
>>>   PASS (0.04 s)
>>>      (6/6) 
>>> /home/travis/build/qemu/qemu/tests/acceptance/vnc.py:Vnc.test_vnc_change_password:
>>>   PASS (0.04 s)
>>>     RESULTS    : PASS 6 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 
>>> | CANCEL 0
>>>     JOB TIME   : 3.90 s
>>>    ...
>>>
>>> Changes from v3:
>>> ================
>>>
>>>  * Fixed typo in commit message (s/requiment/requirement/).  (Eric)
>>>
>>> Changes from v2:
>>> ================
>>>
>>>  * Make the $(TESTS_VENV_DIR) target depend on the
>>>    venv-requirements.txt file, and touch $(TESTS_VENV_DIR) after venv
>>>    runs.  With this, updates on the file are reflected on the
>>>    venv. (Philippe)
>>>
>>>  * Run pip with "python -m pip".  It may have been installed reusing
>>>    the system wide packages, and then the script may not be available
>>>    on the venv. (Philippe)
>>>
>>>  * Dropped Python version on Travis, and using the version supplied
>>>    by the distro (3.4). (Philippe)
>>>
>>>  * Added "python3.4-venv" package requirement on Travis. (Philippe)
>>>
>>>  * Added variable (AVOCADO_SHOW) with logging streams to be shown
>>>    while running the acceptance tests.  By default it's set to none,
>>>    the equivalent of the quiet mode used on previous versions.
>>>    (Philippe)
>>>
>>>  * On Travis, set the AVOCADO_SHOW variable to "app", so that the
>>>    individual test results can be easily seen.  (Philippe)
>>>
>>> Ideas discussed, but not implemented:
>>>
>>>   * Run pip with "$(PYTHON) -m pip -q install ..." because it points
>>>     to the system wide Python installation. (Philippe)
>>>
>>>   * Drop the "--system-site-packages" flag.  Waiting on another round
>>>     of tests to determine if they are really the cause of some package
>>>     installation problems.
>>>
>>> Changes from v1:
>>> ================
>>>
>>>  * TESTS_VENV_REQ (the path of "venv-requirements.txt") now points to
>>>    the source path ($SRC_PATH instead of $BUILD_DIR)
>>>
>>>  * Create the venv with "--system-site-packages", which allows the
>>>    reuse of packages (and no additional downloads) in case there's a
>>>    package installed system wide providing the same package and
>>>    version.
>>>
>>>  * Run Avocado with "python -m avocado".  It may have been installed
>>>    reusing the system wide packages, and then the script may not
>>>    be available on the venv.
>>>
>>>  * Improved documentation describing the Python 3, venv and pip
>>>    requirements.
>>>
>>>  * Updated avocado-framework requirement to latest released version
>>>    (65.0)
>>>
>>>  * (New commit) Added support for running the acceptance tests on
>>>    Travis.
>>>
>>> Ideas discussed, but not implemented:
>>>
>>>  * Install external packages such as python3-pip on Debian based
>>>    systems, deemed too invasive on developer's systems.
>>>
>>>  * Allow the use of Python 2, and consequently the "virtualenv"
>>>    module.
>>>
>>> Cleber Rosa (3):
>>>   Bootstrap Python venv for tests
>>>   Acceptance tests: add make rule for running them
>>>   Travis support for the acceptance tests
>>>
>>>  .travis.yml                 |  5 +++++
>>>  docs/devel/testing.rst      | 35 ++++++++++++++++++++++++++++++-----
>>>  tests/Makefile.include      | 37 +++++++++++++++++++++++++++++++++++++
>>>  tests/venv-requirements.txt |  4 ++++
>>>  4 files changed, 76 insertions(+), 5 deletions(-)
>>>  create mode 100644 tests/venv-requirements.txt
>>>
>>
>