Re: [kvm-devel] [Qemu-devel] [PATCH] QEMU: fsync AIO writes on flush request

[Marcelo Tosatti]

On Sat, Mar 29, 2008 at 01:09:30AM +0000, Jamie Lokier wrote:
> Marcelo Tosatti wrote:
> > I don't think the first qemu_aio_flush() is necessary because the fsync
> > request will be enqueued after pending ones: 
> > 
> >        aio_fsync() function does a sync on all outstanding
> >        asynchronous I/O operations associated with
> >        aiocbp->aio_fildes.
> >
> >        More precisely, if op is O_SYNC, then all currently queued
> >        I/O operations shall be completed as if by a call of
> >        fsync(2), and if op is O_DSYNC, this call is the asynchronous
> >        analog of fdatasync(2).  Note that this is a request only —
> >        this call does not wait for I/O completion.
> > 
> > glibc sets the priority for fsync as 0, which is the same priority AIO
> > reads and writes are submitted by QEMU.
> 
> Do AIO operations always get executed in the order they are submitted?

With glibc AIO on the same file descriptor, yes. See
sysdeps/pthreads/aio_misc.c.

The kernel AIO implementation (used for DIRECT IO) does not implement
aio_fsync.

> I was under the impression this is not guaranteed, as relaxed ordering
> permits better I/O scheduling (e.g. to reduce disk seeks) - which is
> one of the most useful points of AIO.  (Otherwise you might as well
> just have one worker thread doing synchronous IO in order).
> 
> And because of that, I was under the impression the only way to
> implement a write barrier+flush in AIO was (1) wait for pending writes
> to complete, then (2) aio_fsync, then (3) wait for the aio_fsync.
> 
> I could be wrong, but I haven't seen any documentation which says
> otherwise, and it's what I'd expect of an implementation.  I.e. it's
> just an asynchronous version of fsync().

All documentation (and source code) I can find indicates that
aio_fsync() will wait for pending AIO requests to finish before doing
the fsync/fdatasync system call.

I find it hard to see much purpose in such an interface otherwise.

http://www.opengroup.org/onlinepubs/009695399/functions/aio_fsync.html

http://docs.hp.com/en/B9106-90012/aio.5.html "Synchronizing Permanent Storage"

http://www.gnu.org/software/libtool/manual/libc/Synchronizing-AIO-Operations.html#Synchronizing-AIO-Operations

"When dealing with asynchronous operations it is sometimes necessary to
get into a consistent state. This would mean for AIO that one wants to
know whether a certain request or a group of request were processed.
This could be done by waiting for the notification sent by the system
after the operation terminated, but this sometimes would mean wasting
resources (mainly computation time). Instead POSIX.1b defines two
functions which will help with most kinds of consistency."

http://www.governmentsecurity.org/articles/articles2/B2355-90693.pdf_fl/B2355-90693-29.html

This is the HPUX man page, much better than the Linux one.

> The quoted man page doesn't convince me.  It says "all currently
> queued I/O operations shall be completed" which _could_ mean that
> aio_fsync is an AIO barrier too.
> 
> But then "if by a call of fsync(2)" implies that aio_fsync+aio_suspend
> could just be replaced by fsync() with no change of semantics.  So
> "queued I/O operations" means what fsync() handles: dirty file data,
> not in-flight AIO writes.
> 
> And you already noticed that fsync() is _not_ guaranteed to flush
> in-flight AIO writes.  Being the asynchronous analog, aio_fsync()
> would not either.

That seems to be the whole point of aio_fsync().