Re: [Qemu-devel] [RFC v2 2/4] tests: iotests: don't compare SHUTDOWN event

[Peter Xu]

On Mon, Jun 04, 2018 at 12:59:04PM +0800, Peter Xu wrote:
> On Thu, May 31, 2018 at 09:42:23AM -0500, Eric Blake wrote:
> > On 05/31/2018 12:16 AM, Peter Xu wrote:
> > > This event is not really necessary.  After OOB series it might affect
> > > the timing of the script so this event may or may not be there comparing
> > > to the old *.out results.  Let's just filter it out.
> > 
> > This is worrying. Are you stating that the SHUTDOWN event can occur in a
> > different order than it used to, or is it even worse that the SHUTDOWN event
> > disappears altogether?  If enabling OOB makes the SHUTDOWN event sometimes
> > disappear, that's a regression that we should fix.  If it just makes things
> > occur in a different order, we need an explanation why that is okay.
> 
> The event might conditionally disappear in two of the 100+ qcow2
> tests.  And when it happens, it's not disappearing in all the
> testcases in the test but only some.  For example, 087 might
> conditionally fail with this:
> 
> 087 8s ... - output mismatch (see 087.out.bad)
> --- /home/peterx/git/qemu/tests/qemu-iotests/087.out    2018-06-01 
> 18:44:22.378982462 +0800
> +++ /home/peterx/git/qemu/bin/tests/qemu-iotests/087.out.bad    2018-06-01 
> 18:53:44.267840928 +0800
> @@ -8,7 +8,6 @@
>  {"return": {}}
>  {"error": {"class": "GenericError", "desc": "'node-name' must be specified 
> for the root node"}}
>  {"return": {}}
> -{"timestamp": {"seconds":  TIMESTAMP, "microseconds":  TIMESTAMP}, "event": 
> "SHUTDOWN", "data": {"guest": false}}
> 
> 
>  === Duplicate ID ===
> @@ -53,7 +52,6 @@
>  {"return": {}}
>  {"return": {}}
>  {"return": {}}
> -{"timestamp": {"seconds":  TIMESTAMP, "microseconds":  TIMESTAMP}, "event": 
> "SHUTDOWN", "data": {"guest": false}}
> 
> 
>  === Missing driver ===
> 
> Firstly, it does not fail every time I run "./check -qcow2 087", but
> it might fail like 1 out of 5.  Then, it's not failing all the
> testcases in 087.  For above example, it's failing "Missing ID and
> node-name" and "Encrypted image LUKS", and it can change too.

I think I found the problem underneath.  I posted a standalone patch
to fix that here:

  [PATCH] monitor: postpone monitor_qmp_cleanup_queues

Please have a look first on that, then I'll see how I should proceed
with this series.  Thanks,

-- 
Peter Xu