Re: migration qtest failure: "query-migrate shows failed migration: Unable to write to socket: Broken pipe"

[Dr. David Alan Gilbert]

* Peter Maydell (peter.maydell@linaro.org) wrote:
> On Thu, 15 Dec 2022 at 11:40, Dr. David Alan Gilbert
> <dgilbert@redhat.com> wrote:
> >
> > * Peter Maydell (peter.maydell@linaro.org) wrote:
> > > Hi; I see this migration qtest failure on my x86 macos box:
> > >
> > >
> > > ▶  32/591 
> > > ERROR:../../tests/qtest/migration-helpers.c:151:migrate_query_not_failed:
> > > assertion fail
> > > ed: (!g_str_equal(status, "failed")) ERROR
> > >  32/591 qemu:qtest+qtest-aarch64 / qtest-aarch64/migration-test
> > >            ERROR          152.27s   killed by signal 6 SIGABRT
> > > ――――――――――――――――――――――――――――――――――――― ✀  
> > > ―――――――――――――――――――――――――――――――――――――
> > > stderr:
> > > query-migrate shows failed migration: Unable to write to socket: Broken 
> > > pipe
> > > **
> > > ERROR:../../tests/qtest/migration-helpers.c:151:migrate_query_not_failed:
> > > assertion failed: (!g_str_equal(status, "failed"))
> > > Assertion failed: (pid == s->qemu_pid), function qtest_wait_qemu, file
> > > ../../tests/qtest/libqtest.c, line 207.
> > >
> > > (test program exited with status code -6)
> > > ――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――
> > >
> > > and similarly:
> > >
> > > ▶  34/591 
> > > ERROR:../../tests/qtest/migration-helpers.c:151:migrate_query_not_failed:
> > > assertion failed: (!g_str_equal(status, "failed")) ERROR
> > >  34/591 qemu:qtest+qtest-i386 / qtest-i386/migration-test
> > >            ERROR          169.44s   killed by signal 6 SIGABRT
> > > ――――――――――――――――――――――――――――――――――――― ✀  
> > > ―――――――――――――――――――――――――――――――――――――
> > > stderr:
> > > query-migrate shows failed migration: Unable to write to socket: Broken 
> > > pipe
> > > **
> > > ERROR:../../tests/qtest/migration-helpers.c:151:migrate_query_not_failed:
> > > assertion failed: (!g_str_equal(status, "failed"))
> > > Assertion failed: (pid == s->qemu_pid), function qtest_wait_qemu, file
> > > ../../tests/qtest/libqtest.c, line 207.
> > >
> > > (test program exited with status code -6)
> > > ――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――
> > >
> > > It seems to be fairly consistent. Any ideas what it might be?
> > > Maybe the QEMU process has already exited before the test binary
> > > gets round to querying the status ?
> >
> > Yes, it sounds like it, can you get a backtrace to figure out which test
> > it was in/where it was upto when it died?
> 
> The logfile says it had just done
> ok 23 /aarch64/migration/multifd/tcp/plain/none
> so I think the one it was in the middle of when it failed was
> /aarch64/migration/multifd/tcp/plain/cancel.
> Similarly the log suggests the x86 failure was for
> /i386/migration/miltifd/tcp/plain/cancel.
> 
> It doesn't seem to repro running manually, my guess is that
> it happens because the machine is heavily loaded doing the
> whole build-and-test cycle.

Yeh; I think we'll still need a backtrace or better qmp log though to
figure it out.
If I read that correctly, what that test does is:

   start 'from'
   start 'to'
   slow migrate from->to
   <cancel>
   start 'to2'
   migrate from->to2

I'd only expect the 'to' to quit by itself, and I don't think we should
be doing a 'failed' check on a destination (I don't think).
Even if the migration finished quickly 'from' shouldn't just quit - so
the QMP should still be working, so we shouldn't get the broken pipe on
the actual qmp pipe.
Assuming then it's a broken socket on the migration stream, that sounds
like a bug - a 'cancel' might cause the destination to get an error
but the source shouldn't - it should know it's cancelling.

Dave

> thanks
> -- PMM
-- 
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK