Re: [PATCH v2 28/29] migration: Add direct-io parameter

[Daniel P . Berrangé]

On Tue, Oct 31, 2023 at 04:05:46PM -0300, Fabiano Rosas wrote:
> Daniel P. Berrangé <berrange@redhat.com> writes:
> 
> > On Tue, Oct 31, 2023 at 12:52:41PM -0300, Fabiano Rosas wrote:
> >> Daniel P. Berrangé <berrange@redhat.com> writes:
> >> >
> >> > I guess I'm not seeing the problem still.  A single FD is passed across
> >> > from libvirt, but QEMU is free to turn that into *many* FDs for its
> >> > internal use, using dup() and then setting O_DIRECT on as many/few of
> >> > the dup()d FDs as its wants to.
> >> 
> >> The problem is that duplicated FDs share the file status flags. If we
> >> set O_DIRECT on the multifd channels and the main thread happens to do
> >> an unaligned write with qemu_file_put* then the filesystem will fail
> >> that write.
> >
> > Doh, I had forgotten that sharing.
> >
> > Do we have any synchronization between multifd  channels and the main
> > thread ?  eg does the main thread wait for RAM sending completion
> > before carrying on writing other non-RAM data ?
> 
> We do have, but the issue with that approach is that there are no rules
> for adding data into the stream. Anyone could add a qemu_put_* call
> right in the middle of the section for whatever reason.
> 
> That is almost a separate matter due to our current compatibility model
> being based on capabilities rather than resilience of the stream
> format. So extraneous data in the stream always causes the migration to
> break.
> 
> But with the O_DIRECT situation we'd be adding another aspect to
> this. Not only changing the code requires syncing capabilities (as it
> does today), but it would also require knowing which parts of the stream
> can be interrupted by new data and which cannot.
> 
> So while it would probably work, it's also a little fragile. If QEMU
> were given 2 FDs or given access to the file, then only the multifd
> channels would get O_DIRECT and they are guaranteed to not have
> extraneous unaligned data showing up.

So the problem with add-fd is that when requesting a FD, the monitor
code masks flags with O_ACCMODE.  What if we extended it such that
the monitor masked with O_ACCMODE | O_DIRECT.

That would let us pass 1 plain FD and one O_DIRECT fd, and be able
to ask for each separately by setting O_DIRECT or not.

Existing users of add-fd are not likely to be affected since none of
them will be using O_DIRECT.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|