Re: [Qemu-block] [Qemu-devel] Block layer complexity: what to do to keep it under control?

[Stefan Hajnoczi]

On Thu, Nov 30, 2017 at 05:47:09PM +0800, Fam Zheng wrote:
> On Wed, 11/29 12:00, Stefan Hajnoczi wrote:
> > On Wed, Nov 29, 2017 at 11:55:02AM +0800, Fam Zheng wrote:
> > 
> > Event loops and coroutines are good but they should not be used directly
> > by block drivers and block jobs.  We need safe, high-level APIs that
> > implement commonly-used operations.
> > 
> > > - Documentation
> > > 
> > >   There is no central developer doc about block layer, especially how all 
> > > pieces
> > >   fit together. Having one will make it a lot easier for new contributors 
> > > to
> > >   understand better. Of course, we're facing the old problem: the code is
> > >   moving, maintaining an updated document needs effort.
> > > 
> > >   Idea: add ./doc/deve/block.txt?
> > 
> > IOThreads and AioContexts are addressed here:
> > docs/devel/multiple-iothreads.txt
> > 
> > The game has become significantly more complex than what the document
> > describes.  It's lacking aio_co_wake() and aio_co_schedule() for
> > example.
> > 
> > > - Simplified code, or more orthogonal/modularized architecture.
> > > 
> > >   Each aspect of block layer is complex enough so isolating them as much 
> > > as
> > >   possible is a reasonable approach to control the complexity. Block jobs 
> > > and
> > >   throttling becoming block filters is a good example, we should identify 
> > > more.
> > > 
> > >   Idea: rethink event loops. Create coroutines ubiquitously (for example 
> > > for
> > >   each fd handler, BH and timer), so that many nested aio_poll() can be 
> > > removed.
> > > 
> > >   Crazy idea: move the whole block layer to a vhost process, and implement
> > >   existing features differently, especially in terms of multi-threading 
> > > (hint:
> > >   rust?).
> > 
> > A reimplementation will not solve the problem because:
> > 
> > 1. If it still has the same feature set and requirements then the level
> >    of complexity will be comparable.
> > 
> > 2. We can reduce accidental (inessential) complexity by continuing the
> >    various efforts around the block graph, block jobs, multi-queue block
> >    layer with an eye towards higher level APIs.
> 
> Starting over is certainly not the motivation to do qemu-vhost, but it would 
> be
> an opportunity to use different async/concurrency paradigms if that is going 
> to
> happen. I think in current block layer, event loop + coroutine is a good
> combination, but having nested aio_poll()'s made it worse, then mixing 
> IOThreads
> in makes it a lot more complicated.

Why alternative model are you thinking of?

One slight change is to make everything run in a coroutine so that there
are no while (aio_poll()) loops.  Instead the caller would yield.

But the fundamental problem remains that drain is necessary and the
storage stack does not support request cancellation.  For example, when
the virtio-blk-pci device is reset QEMU has no way of (safely)
cancelling requests so it has to wait for all requests to complete.
This means we're stuck with synchronization points.

I agree that adding threading makes thing more complicated but
ultimately there is a real requirement to do that.  It's like the
difference between a simple block driver that is designed only for
qemu-img convert versus a fully-featured block driver that supports
parallel I/O.  The complexity is much higher but you can't wish it away
if you want parallel I/O.

Stefan

signature.asc
Description: PGP signature