Re: [Qemu-block] Block layer complexity: what to do to keep it under control?

[Jeff Cody]

On Wed, Nov 29, 2017 at 11:55:02AM +0800, Fam Zheng wrote:
> Hi all,
> 
> As we move forwards with new features in the block layer, the chances of 
> tricky
> bugs happening have been increasing alongside - block jobs, coroutines,
> throttling, AioContext, op blockers and image locking combined together make a
> large and complex picture that is hard to fully understand and work with. Some
> bugs we've encountered are quite challenging already.  Examples are:
> 
> - segfault in parallel blockjobs (iotest 30)
>   https://lists.gnu.org/archive/html/qemu-devel/2017-11/msg01144.html
> 
> - Intermittent hang of iotest 194 (bdrv_drain_all after non-shared storage
>   migration)
>   https://lists.gnu.org/archive/html/qemu-devel/2017-11/msg01626.html
> 
> - Drainage in bdrv_replace_child_noperm()
>   https://lists.gnu.org/archive/html/qemu-devel/2017-11/msg00868.html
> 
> - Regression from 2.8: stuck in bdrv_drain()
>   https://lists.gnu.org/archive/html/qemu-devel/2017-04/msg02193.html
> 

I agree, it seems the complexity is growing by quite a bit.

> So in principle, what should we do to make the block layer easy to understand,
> develop with and debug? I think we have opportunities in these aspects:
> 
> - Documentation
> 
>   There is no central developer doc about block layer, especially how all 
> pieces
>   fit together. Having one will make it a lot easier for new contributors to
>   understand better. Of course, we're facing the old problem: the code is
>   moving, maintaining an updated document needs effort.
> 
>   Idea: add ./doc/deve/block.txt?
> 

There are some bits of brilliance in what is already there; for instance,
devel/atomics.txt is very thorough.  But I agree that a major piece missing
is an overall design document, that provides the "why" to the "what".

Even given the cost of maintaining a higher level design document, I
think your suggestion here is probably the one that can help mitigate the
complexity the most; the more we (developers) can keep a coherent design
model in mind, the better we are able to do your _other_ suggestions: create
effective tests, simplify code, and enhance debuggability.

> - Tests
> 
>   Writing tests is a great way not only to exercise code, verify new features
>   work as expected and catch regression bugs, but also a way to show how the
>   feature can be used. There is this trend that the QEMU user interface
>   gradually moves from high level commands and args to small and flexible
>   building blocks, therefore demostrating the usage in iotests is meaningful.
> 
>   Idea: Add tests to simulate how libvirt uses block layer, or how we expect 
> it
>   to. This would be a long term investment. We could reuse iotests, or create 
> a
>   new test framework specifically, if it's easier (for example, use docker/vm
>   tests that just uses libvirt).
> 
>   Idea: Patchew already tests the quick group of iotests for a few
>   formats/protocols, but we should really add it to "make check".
> 

Perhaps higher level testing (like your example of how libvirt uses the
block layer) is a good candidate for avocado?


> - Simplified code, or more orthogonal/modularized architecture.
> 
>   Each aspect of block layer is complex enough so isolating them as much as
>   possible is a reasonable approach to control the complexity. Block jobs and
>   throttling becoming block filters is a good example, we should identify 
> more.
> 
>   Idea: rethink event loops. Create coroutines ubiquitously (for example for
>   each fd handler, BH and timer), so that many nested aio_poll() can be 
> removed.
> 
>   Crazy idea: move the whole block layer to a vhost process, and implement
>   existing features differently, especially in terms of multi-threading (hint:
>   rust?).
> 
> - Debuggability.
> 
>   Working with backtraces when coroutine is used is pretty hard, it would be
>   nice if ./scripts/qemugdb/coroutine.py could work with core files (i.e.
>   without a process to debug), and trace back to co->caller automatically.
> 

IIRC, this used to work, right?

>   It's always useful to dump block graph. Maybe we should add a helper 
> function
>   in block layer that dumps all node graphs in graphviz DOT format, and even
>   make it available in QMP as x-dump-block-graph?
> 
>   Of course gdb scripts to dump various lists are also nice little things to
>   have.
> 
>   Idea: write more ./scripts/qemugdb/<scriptlet>.py.

More qemugdb macros would be great, especially for dumping the block chain
and making coroutines less opaque.

-Jeff