Re: [PATCH 1/2] python: introduce qmp-shell-wrap convenience tool

[Daniel P . Berrangé]

On Thu, Jan 20, 2022 at 02:33:46PM +0100, Philippe Mathieu-Daudé wrote:
> On 18/1/22 19:04, John Snow wrote:
> > On Tue, Jan 18, 2022 at 5:06 AM Daniel P. Berrangé <berrange@redhat.com> 
> > wrote:
> 
> > > It would be nice to just have this integrated into 'make check' so we
> > > don't need to remember to run a special command.
> > 
> > The CI will run it, but 'make check' doesn't. To add it to make check,
> > I need to figure out how to insert a venv-building step into 'make
> > check' such that the venv gets deposited into the build dir instead of
> > the source dir.
> > I think I may also need yet another set of package dependencies that
> > pin on precise dependencies for testing purposes to prevent random
> > regressions during 'make check' when nobody has touched the Python
> > code.
> > 
> > Overall, I felt like maybe it was more hassle than it was worth if I
> > can just nudge people touching the python to run a 'make check-dev'
> > every so often.
> > 
> > Patches welcome, etc. My overall strategy with the python tests so far has 
> > been:
> > 
> > (1) Keep python tests fully separate from the QEMU build system to
> > allow them to be split out into new repositories easily.
> > (2) Use the pipenv test to lock the very oldest dependencies the code
> > and tests support, using the very oldest python we support (3.6) This
> > test is used as the gating test in GitLab CI, as it is very repeatable
> > and the GitLab CI setup ensures I can always have the exact Python
> > packages it requires available.
> > (3) Use the tox test to test against a wide variety of Python
> > interpreters (3.6 through 3.10 inclusive) using the very latest python
> > packages to detect regressions on cutting-edge environments
> > (4) Use the widest possible range of versions for dependent packages
> > in setup.cfg such that QEMU packages are unlikely to cause versioning
> > conflicts in environments that decide to integrate our code.
> > 
> > Overall, I test on 3.6 through 3.10, and against the "oldest" and
> > "newest" dependencies. It's a good, wide matrix.
> > 
> > However, It's #4 there that runs me into trouble with tests that are
> > guaranteed to pass -- the linters update all the time and cause new
> > problems. I use pipenv to lock to specific versions, but that tool
> > wants to run against Python 3.6 *explicitly*, so it isn't suitable for
> > a generic purpose 'make check' because not everyone will have a Python
> > 3.6 interpreter available. I need something kind of halfway between,
> > where I can lock against specific versions but not against the Python
> > interpreter version, and that's what could be used for a decent 'make
> > check' test.
> > 
> > Of course, I don't want to maintain like 10 versions of a dependent
> > packages list, either.
> > 
> > (I really, really wish pip had an --use-oldest flag for dependency
> > resolution, but it doesn't.)
> 
> Could we simply use a virtualenv for all QEMU testing tasks (packages
> consumed by QEMU tests), and only deal with installed Python packages
> for regular non-testing QEMU uses (things exposed via pyqemu that we
> want stable)?

I don't especially like the idea of using virtualenv. It is a good thing
that we're testing on the distro python packages, because that is the
scenario that our contributors/developers will actually use these
tools in. We're got a well defined set of target platforms that QEMU
aims for that we need to cover in testing. If we also want to do tests
against general python using a virtualenv in CI pipelines thats fine,
but doesn't replace the need to testing against our explicit platform
targets, its just additive.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|