Re: [PULL 22/33] migration: remove the QEMUFileOps 'get_buffer' callback

[Daniel P . Berrangé]

On Mon, Jun 27, 2022 at 04:32:00PM -0400, Peter Xu wrote:
> On Mon, Jun 27, 2022 at 04:03:09PM +0100, Daniel P. Berrangé wrote:
> > On Wed, Jun 22, 2022 at 03:34:52PM -0400, Peter Xu wrote:
> > > On Wed, Jun 22, 2022 at 07:39:06PM +0100, Dr. David Alan Gilbert (git) 
> > > wrote:
> > > > diff --git a/migration/qemu-file.c b/migration/qemu-file.c
> > > > index 74f919de67..e206b05550 100644
> > > > --- a/migration/qemu-file.c
> > > > +++ b/migration/qemu-file.c
> > > > @@ -377,8 +377,22 @@ static ssize_t qemu_fill_buffer(QEMUFile *f)
> > > >          return 0;
> > > >      }
> > > >  
> > > > -    len = f->ops->get_buffer(f->ioc, f->buf + pending, 
> > > > f->total_transferred,
> > > > -                             IO_BUF_SIZE - pending, &local_error);
> > > > +    do {
> > > > +        len = qio_channel_read(f->ioc,
> > > > +                               (char *)f->buf + pending,
> > > > +                               IO_BUF_SIZE - pending,
> > > > +                               &local_error);
> > > > +        if (len == QIO_CHANNEL_ERR_BLOCK) {
> > > > +            if (qemu_in_coroutine()) {
> > > > +                qio_channel_yield(f->ioc, G_IO_IN);
> > > > +            } else {
> > > > +                qio_channel_wait(f->ioc, G_IO_IN);
> > > > +            }
> > > > +        } else if (len < 0) {
> > > > +            len = EIO;
> > > 
> > > This should be -EIO.
> > > 
> > > > +        }
> > > > +    } while (len == QIO_CHANNEL_ERR_BLOCK);
> > > 
> > > It's failing only with the new TLS test I added for postcopy somehow (at
> > > least /x86_64/migration/postcopy/recovery/tls).. I also verified after the
> > > change it'll work again.
> > 
> > Assuming you can still reproduce the pre-existing flaw, can you capture
> > a stack trace when it hangs.   I'm wondering if it is a sign that the
> > migration is not converging when using TLS under certain load conditions,
> > because the test waits forever for converge.
> 
> Yes it is, and it reproduces here every time.  It hangs at:
> 
>  if (!got_stop) {
>      qtest_qmp_eventwait(from, "STOP");
>  }
> 
> > 
> > Also what scenario are you running in ? Bare metal or a VM, and what
> > host arch ? Wondering if the machine is at all slow, or for example
> > missing AES hardware acceleration or some such thing.
> 
> It's Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz, 40 cores.
> 
> It'll pass after I modify the downtime:
> 
>   migrate_set_parameter_int(from, "downtime-limit", 100000);
> 
> And with QTEST_LOG=1 I found that the bw is indeed low, ~700mbps.

Good, this all makes sense, and I've got pending patchues I'm testing
that will fix this.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|