Re: [RFC PATCH 4/8] tests/pytest: add pytest to the meson build system

[Thomas Huth]

On 12/07/2024 13.47, Daniel P. Berrangé wrote:
> On Fri, Jul 12, 2024 at 12:14:45PM +0200, Thomas Huth wrote:
> > On 12/07/2024 11.01, Daniel P. Berrangé wrote:
> > > On Thu, Jul 11, 2024 at 01:55:42PM +0200, Thomas Huth wrote:
> > > > From: Ani Sinha <ani@anisinha.ca>
> > > >
> > > > Integrate the pytest framework with the meson build system. This
> > > > will make meson run all the pytests under the pytest directory.
> > >
> > > Lets add a note about the compelling benefit of this new approach
> > >
> > >     With this change, each functional test becomes subject
> > >     to an individual execution timeout, defaulting to 60
> > >     seconds, but overridable per-test.
> >
> > The avocado runner uses timeouts, too, so it's not really an additional
> > benefit that we get here.
> >
> > > For CI purposes we'll need to add 'python3-pytest' to
> > > tests/lcitool/projects/qemu.yml, and re-generate the
> > > the dockerfiles. Some of the other non-gitlab CI
> > > integrations probably need manual additions of pytest
> > > packages.
> >
> > I'm currently rather looking into getting rid of pytest and to use pycotap
> > instead: Using the TAP protocol for running the tests, you get a much nicer
> > output from the meson test runner, which can then count the subtests and
> > properly report SKIPs for tests that have not been run.
>
> I've just looked at pycotap and IIUC, there's no command line
> tool equivalent to '/usr/bin/pytest' at all. Each test case
> is expected to provide a stub for "__main__" to invoke the
> tests.

With pycotap 1.3.0 you could run tests through "python -m pycotap ...", too. 
But for playing nicely with meson, I'd like to set message_log and 
test_output_log to LogMode.LogToError instead of using their default value, 
so I need my own wrapper code anyway.

> As such each individual test is directly executable.
> This meshes nicely with what I'd suggested as changes in
> patch 1, and eliminating the intermediate runner process is
> a nice further simplification. So I'll be interested to see
> your next version using pycotap.

I'll try to post it as soon as I have something at least in a somewhat more 
proper shape than the RFC ;-)

 Thomas