Re: [kvm-devel] [Qemu-devel] Re: [PATCH 1/3] Refactor AIO interface to allow other AIO implementations

[Avi Kivity]

Jamie Lokier wrote:
> Avi Kivity wrote:
>   
> > > At such a tiny difference, I'm wondering why Linux-AIO exists at all,
> > > as it complicates the kernel rather a lot.  I can see the theoretical
> > > appeal, but if performance is so marginal, I'm surprised it's in
> > > there.
> > >       
> > Linux aio exists, but that's all that can be said for it.  It works 
> > mostly for raw disks, doesn't integrate with networking, and doesn't 
> > advance at the same pace as the rest of the kernel.  I believe only 
> > databases use it (and a userspace filesystem I wrote some time ago).
> >     
>
> And video streaming on some embedded devices with no MMU!  (Due to the
> page cache heuristics working poorly with no MMU, sustained reliable
> streaming is managed with O_DIRECT and the app managing cache itself
> (like a database), and that needs AIO to keep the request queue busy.
> At least, that's the theory.)
>
>   

Could use threads as well, no?

> > > I'm also surprised the Glibc implementation of AIO using ordinary
> > > threads is so close to it.  
> > >       
> > Why are you surprised?
> >     
>
> Because I've read that Glibc AIO (which uses a thread pool) is a
> relatively poor performer as AIO implementations go, and is only there
> for API compatibility, not suggested for performance.
>
> But I read that quite a while ago, perhaps it's changed.
>
>   

It's me at fault here.  I just assumed that because it's easy to do aio 
in a thread pool efficiently, that's what glibc does.

Unfortunately the code does some ridiculous things like not service 
multiple requests on a single fd in parallel.  I see absolutely no 
reason for it (the code says "fight for resources").

So my comments only apply to linux-aio vs a sane thread pool.  Sorry for 
spreading confusion.

> > Actually the glibc implementation could be improved from what I've 
> > heard.  My estimates are for a thread pool implementation, but there is 
> > not reason why glibc couldn't achieve exactly the same performance.
> >     
>
> Erm...  I thought you said it _does_ achieve nearly the same
> performance, not that it _could_.
>
> Do you mean it could achieve exactly the same performance by using
> Linux AIO when possible?
>
>   

It could and should.  It probably doesn't.

A simple thread pool implementation could come within 10% of Linux aio 
for most workloads.  It will never be "exactly", but for small numbers 
of disks, close enough.

> > > And then, I'm wondering why use AIO it
> > > all: it suggests QEMU would run about as fast doing synchronous I/O in
> > > a few dedicated I/O threads.
> > >       
> > Posix aio is the unix API for this, why not use it?
> >     
>
> Because far more host platforms have threads than have POSIX AIO.  (I
> suspect both options will end up supported in the end, as dedicated
> I/O threads were already suggested for other things.)
>   

Agree.

>   
> > > > Also, I'd presume that those that need 10K IOPS and above will not place 
> > > > their high throughput images on a filesystem; rather on a separate SAN 
> > > > LUN.
> > > >         
> > > Does the separate LUN make any difference?  I thought O_DIRECT on a
> > > filesystem was meant to be pretty close to block device performance.
> > >       
> > On a good extent-based filesystem like XFS you will get good performance 
> > (though more cpu overhead due to needing to go through additional 
> > mapping layers.  Old clunkers like ext3 will require additional seeks or 
> > a ton of cache (1 GB per 1 TB).
> >     
>
> Hmm.  Thanks.  I may consider switching to XFS now....
>
>   

I'm rooting for btrfs myself.

--
error compiling committee.c: too many arguments to function