Re: [PATCH] Revert "check-block: replace -makecheck with TAP output"

[Daniel P . Berrangé]

On Tue, Feb 08, 2022 at 07:44:32AM -0500, Paolo Bonzini wrote:
> This reverts commit d316859f4e28c74ab8b618895d2a5e0a865d3cf1.  The
> TAP output is inferior in that it does not include the diff for
> failed tests.
> 
> Reported-by: Peter Maydell <peter.maydell@linaro.org>
> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
> ---
>  tests/check-block.sh             |  6 ++--
>  tests/qemu-iotests/check         |  6 ++--
>  tests/qemu-iotests/meson.build   |  1 -
>  tests/qemu-iotests/testenv.py    | 30 +++++++++----------
>  tests/qemu-iotests/testrunner.py | 49 +++++++++++++++-----------------
>  5 files changed, 44 insertions(+), 48 deletions(-)

Peter mentioned that he's seeing the CentOS 8 CI job fail
continuously: eg

https://gitlab.com/qemu-project/qemu/-/jobs/2163919781

[quote]
1/1 qemu:block / qemu-iotests qcow2 ERROR          163.04s   120 subtests passed
Summary of Failures:
1/1 qemu:block / qemu-iotests qcow2 ERROR          163.04s   120 subtests passed
Ok:                 0   
Expected Fail:      0   
Fail:               1   
Unexpected Pass:    0   
Skipped:            0   
Timeout:            0   
Full log written to 
/home/gitlab-runner/builds/Jpwtyaz7/0/qemu-project/qemu/build/meson-logs/iotestslog.txt
make: *** 
[/home/gitlab-runner/builds/Jpwtyaz7/0/qemu-project/qemu/tests/Makefile.include:156:
 check-block] Error 1

[/quote]

None of the individual tests are showing ERROR though, only the final
status is error.

We also don't have access to iotestslog.txt which is very annoying.

When i run it locally I got the iotests.log and found that it
contained an error message saying the TAP outout was malformed
and couldn't be parsed.

When I ran the iotests manually I confirmed this, it prints:

ok qcow2 037
ok qcow2 038
ok qcow2 039
ok qcow2 040
ok qcow2 041
    [case not run] test_after_a_quorum_snapshot (__main__.TestRepairQuorum): 
formats ['quorum'] are not whitelisted
    [case not run] test_cancel (__main__.TestRepairQuorum): formats ['quorum'] 
are not whitelisted
    [case not run] test_cancel_after_ready (__main__.TestRepairQuorum): formats 
['quorum'] are not whitelisted
    [case not run] test_complete (__main__.TestRepairQuorum): formats 
['quorum'] are not whitelisted
    [case not run] test_device_not_found (__main__.TestRepairQuorum): formats 
['quorum'] are not whitelisted
    [case not run] test_image_not_found (__main__.TestRepairQuorum): formats 
['quorum'] are not whitelisted
    [case not run] test_medium_not_found (__main__.TestRepairQuorum): formats 
['quorum'] are not whitelisted
    [case not run] test_no_node_name (__main__.TestRepairQuorum): formats 
['quorum'] are not whitelisted
    [case not run] test_nonexistent_replaces (__main__.TestRepairQuorum): 
formats ['quorum'] are not whitelisted
    [case not run] test_pause (__main__.TestRepairQuorum): formats ['quorum'] 
are not whitelisted
    [case not run] test_with_other_parent (__main__.TestRepairQuorum): formats 
['quorum'] are not whitelisted
    [case not run] test_with_other_parents_after_mirror_start 
(__main__.TestRepairQuorum): formats ['quorum'] are not whitelisted
    [case not run] test_wrong_sync_mode (__main__.TestRepairQuorum): formats 
['quorum'] are not whitelisted

ok qcow2 042
ok qcow2 043
ok qcow2 046


All those '[case not run]' lines are invalid tap output.  The
need to start with a leading '#' to be treated as a diagnostic
message. I briefly considered sending a patch, until I remember
this change to stop using TAP.

Anyway I'd say the commit message here needs updating. It is
not merely improving diff output. It is adressing the fact that
the I/O tests runner is not producing valid TAP output, and so
causing bogus failures.


Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|