[Qemu-devel] [PATCH 00/16] aio_context_acquire/release pushdown, part 2

[Paolo Bonzini]

This series pushes down aio_context_acquire/release to the point
where we can actually reason on using different fine-grained mutexes.

The main infrastructure is introduced in patch 1.  The new API aio_co_wake
starts a coroutine with aio_context_acquire/release protection, which
requires tracking each coroutine's "home" AioContext.  aio_co_schedule
instead takes care of moving a sleeping coroutine to a different
AioContext, also ensuring that it runs under aio_context_acquire/release.
This is useful to implement bdrv_set_aio_context, as a simpler alternative
to bottom halves.  Even though one-shot BHs are already simpler than
what we had before, after this patch aio_co_wake and aio_co_schedule
save you from having to do aio_context_acquire/release explicitly.

After patch 2 and 3, which are just small preparatory changes, patches
4 to 7 provide an example of how to use the new API.  In particular patch
4 to 6 implement a new organization of coroutines in the NBD client,
which allows not blocking on partial reply header reads.

Patch 8 introduces helpers for AioContext locking in QED, which is
the most complex AIO-based driver left.  Then the actual meat of the
series runs from patch 9 to patch 13, followed by small optimizations
in patches 14 and 15.

The patches do some back and forth in adding/removing
aio_context_acquire/release calls in block/*.c but ultimately a small
number of aio_context_acquire/release pairs are added after the pushdown.
These are mostly in drivers that use external libraries (where they
actually could already be replaced by QemuMutex) and in device models.

Notably, coroutines need not care about aio_context_acquire/release.
The device models ensure that the first creation of the coroutine has
the AioContext, while aio_co_wake/aio_co_schedule do the same after
they yield.  Therefore, most of the files only need to use those two
functions instead of, respectively, qemu_coroutine_enter and
aio_bh_schedule_oneshot.

However, this is only an intermediate step which is needed because the
block layer and qemu-coroutine locks are thread-unsafe.  So the next
part will add separate locking, independent of AioContext, to block.c and
mostly block/io.c---this includes making CoMutex thread-safe.  Patch 16
therefore already documents the current locking policies block.h to
prepare for the next series.

Paolo

Paolo Bonzini (16):
  aio: introduce aio_co_schedule and aio_co_wake
  block-backend: allow blk_prw from coroutine context
  test-thread-pool: use generic AioContext infrastructure
  io: add methods to set I/O handlers on AioContext
  io: make qio_channel_yield aware of AioContexts
  nbd: do not block on partial reply header reads
  coroutine-lock: reschedule coroutine on the AioContext it was running
    on
  qed: introduce qed_aio_start_io and qed_aio_next_io_cb
  aio: push aio_context_acquire/release down to dispatching
  block: explicitly acquire aiocontext in timers that need it
  block: explicitly acquire aiocontext in callbacks that need it
  block: explicitly acquire aiocontext in bottom halves that need it
  block: explicitly acquire aiocontext in aio callbacks that need it
  aio-posix: partially inline aio_dispatch into aio_poll
  async: remove unnecessary inc/dec pairs
  block: document fields protected by AioContext lock

 aio-posix.c                    |  60 +++---------
 aio-win32.c                    |  30 ++----
 async.c                        |  81 ++++++++++++++--
 block/blkdebug.c               |   9 +-
 block/blkreplay.c              |   2 +-
 block/block-backend.c          |  13 ++-
 block/curl.c                   |  44 ++++++---
 block/gluster.c                |   9 +-
 block/io.c                     |   4 +-
 block/iscsi.c                  |  15 ++-
 block/linux-aio.c              |  10 +-
 block/mirror.c                 |  12 ++-
 block/nbd-client.c             | 108 ++++++++-------------
 block/nbd-client.h             |   2 +-
 block/nfs.c                    |   9 +-
 block/qed-cluster.c            |   2 +
 block/qed-table.c              |  12 ++-
 block/qed.c                    |  58 +++++++----
 block/qed.h                    |   3 +
 block/sheepdog.c               |  29 +++---
 block/ssh.c                    |  29 ++----
 block/throttle-groups.c        |   2 +
 block/win32-aio.c              |   9 +-
 dma-helpers.c                  |   2 +
 hw/block/virtio-blk.c          |  19 +++-
 hw/scsi/scsi-bus.c             |   2 +
 hw/scsi/scsi-disk.c            |  15 +++
 hw/scsi/scsi-generic.c         |  20 +++-
 hw/scsi/virtio-scsi.c          |   6 ++
 include/block/aio.h            |  38 +++++++-
 include/block/block_int.h      |  64 ++++++++-----
 include/io/channel.h           |  59 +++++++++++-
 include/qemu/coroutine_int.h   |  10 +-
 include/sysemu/block-backend.h |  14 ++-
 io/channel-command.c           |  13 +++
 io/channel-file.c              |  11 +++
 io/channel-socket.c            |  16 +++-
 io/channel-tls.c               |  12 +++
 io/channel-watch.c             |   6 ++
 io/channel.c                   |  97 +++++++++++++++----
 nbd/client.c                   |   2 +-
 nbd/common.c                   |   9 +-
 nbd/server.c                   |   4 +
 tests/Makefile.include         |  15 ++-
 tests/iothread.c               |  91 ++++++++++++++++++
 tests/iothread.h               |  25 +++++
 tests/test-aio-multithread.c   | 213 +++++++++++++++++++++++++++++++++++++++++
 tests/test-thread-pool.c       |  12 +--
 tests/test-vmstate.c           |  11 ---
 thread-pool.c                  |   6 +-
 trace-events                   |   4 +
 util/qemu-coroutine-lock.c     |   5 +-
 util/qemu-coroutine-sleep.c    |   2 +-
 util/qemu-coroutine.c          |   8 ++
 util/trace-events              |   1 -
 55 files changed, 1012 insertions(+), 352 deletions(-)
 create mode 100644 tests/iothread.c
 create mode 100644 tests/iothread.h
 create mode 100644 tests/test-aio-multithread.c

-- 
2.9.3