[Qemu-devel] Block layer complexity: what to do to keep it under control?

[Fam Zheng]

Hi all,

As we move forwards with new features in the block layer, the chances of tricky
bugs happening have been increasing alongside - block jobs, coroutines,
throttling, AioContext, op blockers and image locking combined together make a
large and complex picture that is hard to fully understand and work with. Some
bugs we've encountered are quite challenging already.  Examples are:

- segfault in parallel blockjobs (iotest 30)
  https://lists.gnu.org/archive/html/qemu-devel/2017-11/msg01144.html

- Intermittent hang of iotest 194 (bdrv_drain_all after non-shared storage
  migration)
  https://lists.gnu.org/archive/html/qemu-devel/2017-11/msg01626.html

- Drainage in bdrv_replace_child_noperm()
  https://lists.gnu.org/archive/html/qemu-devel/2017-11/msg00868.html

- Regression from 2.8: stuck in bdrv_drain()
  https://lists.gnu.org/archive/html/qemu-devel/2017-04/msg02193.html

So in principle, what should we do to make the block layer easy to understand,
develop with and debug? I think we have opportunities in these aspects:

- Documentation

  There is no central developer doc about block layer, especially how all pieces
  fit together. Having one will make it a lot easier for new contributors to
  understand better. Of course, we're facing the old problem: the code is
  moving, maintaining an updated document needs effort.

  Idea: add ./doc/deve/block.txt?

- Tests

  Writing tests is a great way not only to exercise code, verify new features
  work as expected and catch regression bugs, but also a way to show how the
  feature can be used. There is this trend that the QEMU user interface
  gradually moves from high level commands and args to small and flexible
  building blocks, therefore demostrating the usage in iotests is meaningful.

  Idea: Add tests to simulate how libvirt uses block layer, or how we expect it
  to. This would be a long term investment. We could reuse iotests, or create a
  new test framework specifically, if it's easier (for example, use docker/vm
  tests that just uses libvirt).

  Idea: Patchew already tests the quick group of iotests for a few
  formats/protocols, but we should really add it to "make check".

- Simplified code, or more orthogonal/modularized architecture.

  Each aspect of block layer is complex enough so isolating them as much as
  possible is a reasonable approach to control the complexity. Block jobs and
  throttling becoming block filters is a good example, we should identify more.

  Idea: rethink event loops. Create coroutines ubiquitously (for example for
  each fd handler, BH and timer), so that many nested aio_poll() can be removed.

  Crazy idea: move the whole block layer to a vhost process, and implement
  existing features differently, especially in terms of multi-threading (hint:
  rust?).

- Debuggability.

  Working with backtraces when coroutine is used is pretty hard, it would be
  nice if ./scripts/qemugdb/coroutine.py could work with core files (i.e.
  without a process to debug), and trace back to co->caller automatically.

  It's always useful to dump block graph. Maybe we should add a helper function
  in block layer that dumps all node graphs in graphviz DOT format, and even
  make it available in QMP as x-dump-block-graph?

  Of course gdb scripts to dump various lists are also nice little things to
  have.

  Idea: write more ./scripts/qemugdb/<scriptlet>.py.

More thoughts?

Fam