Re: [Qemu-devel] Improving QMP test coverage

[Cleber Rosa]

On 07/21/2017 11:33 AM, Stefan Hajnoczi wrote:
>> Output testing style delegates checking ouput to diff.  I rather like it
>> when text output is readily available.  It is when testing QMP.  A
>> non-trivial example using this style could be useful, as discussing
>> ideas tends to be more productive when they come with patches.
> 
> Yes, I was considering how many of the Python iotests could be rewritten
> comfortably in shell.  It is nice when the test simply executes commands
> and the output file shows the entire history of what happened.  Great
> for debugging.
> 
> Stefan
> 
I'd like to have a better understanding of the major pain points here.

Although this can be seen as a matter of taste, style preferences and
even religion, I guess it's safe to say that Python can scale better
than shell.  The upside of shell based tests is the "automatic" and
complete logging, right?  Running "bash -x /path/to/test.sh" will give
much more *useful* information than "python -v /path/to/test.py" will, fact.

I believe this has to do with how *generic* Python code is written, and
how builtin functions and most of the standard Python libraries work as
they do.  Now, when writing code aimed at testing, making use of testing
oriented libraries and tools, one would expect much more useful and
readily available debug information.

I'm biased, for sure, but that's what you get when you write basic tests
using the Avocado libraries.  For instance, when using process.run()[1]
within a test, you can choose to see its command output quite easily
with a command such as "avocado --show=avocado.test.stdout run test.py".

Using other custom logging channels is also trivial (for instance for
specific QMP communication)[2][3].

I wonder if such logging capabilities fill in the gap of what you
describe as "[when the] output file shows the entire history of what
happened".

BTW, I'll defer the discussion of using an external tool to check the
output and determine test success/failure, because it is IMO a
complementary topic, and I believe I understand its use cases.

Regards,
- Cleber.

[1] -
http://avocado-framework.readthedocs.io/en/52.0/api/utils/avocado.utils.html#avocado.utils.process.run
[2] -
http://avocado-framework.readthedocs.io/en/52.0/WritingTests.html#advanced-logging-capabilities
[3] - https://www.youtube.com/watch?v=htUbOsh8MZI

-- 
Cleber Rosa
[ Sr Software Engineer - Virtualization Team - Red Hat ]
[ Avocado Test Framework - avocado-framework.github.io ]
[  7ABB 96EB 8B46 B94D 5E0F  E9BB 657E 8D33 A5F2 09F3  ]

signature.asc
Description: OpenPGP digital signature