Re: [Qemu-devel] Improving QMP test coverage

[Daniel P. Berrange]

On Mon, Jul 24, 2017 at 08:56:50AM +0200, Markus Armbruster wrote:
> Cleber Rosa <address@hidden> writes:
> 
> > On 07/21/2017 11:33 AM, Stefan Hajnoczi wrote:
> >>> Output testing style delegates checking ouput to diff.  I rather like it
> >>> when text output is readily available.  It is when testing QMP.  A
> >>> non-trivial example using this style could be useful, as discussing
> >>> ideas tends to be more productive when they come with patches.
> >> 
> >> Yes, I was considering how many of the Python iotests could be rewritten
> >> comfortably in shell.  It is nice when the test simply executes commands
> >> and the output file shows the entire history of what happened.  Great
> >> for debugging.
> >> 
> >> Stefan
> >> 
> > I'd like to have a better understanding of the major pain points here.
> >
> > Although this can be seen as a matter of taste, style preferences and
> > even religion, I guess it's safe to say that Python can scale better
> > than shell.  The upside of shell based tests is the "automatic" and
> > complete logging, right?  Running "bash -x /path/to/test.sh" will give
> > much more *useful* information than "python -v /path/to/test.py" will, fact.
> >
> > I believe this has to do with how *generic* Python code is written, and
> > how builtin functions and most of the standard Python libraries work as
> > they do.  Now, when writing code aimed at testing, making use of testing
> > oriented libraries and tools, one would expect much more useful and
> > readily available debug information.
> >
> > I'm biased, for sure, but that's what you get when you write basic tests
> > using the Avocado libraries.  For instance, when using process.run()[1]
> > within a test, you can choose to see its command output quite easily
> > with a command such as "avocado --show=avocado.test.stdout run test.py".
> >
> > Using other custom logging channels is also trivial (for instance for
> > specific QMP communication)[2][3].
> >
> > I wonder if such logging capabilities fill in the gap of what you
> > describe as "[when the] output file shows the entire history of what
> > happened".
> 
> Test code language is orthogonal to verification method (with code
> vs. with diff).  Except verifying with shell code would be obviously
> nuts[*].
> 
> The existing iotests written in Python verify with code, and the ones
> written in shell verify with diff.  Doesn't mean that we have to port
> from Python to shell to gain "verify with diff".

Nb, not all the python tests verify with code. The LUKS test 149 that
I wrote in python verifies  with diff. I chose python because shell is
an awful programming language if the code needs conditionals, non-scalar
data structures, or is more than 10 lines long in total :-)

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|