Re: [Qemu-devel] [RFC] dataplane: IOThreads and writing dataplane-capable code

[Fam Zheng]

Great document, thanks for writing this! I have a few questions below.

On Thu, 05/08 12:16, Stefan Hajnoczi wrote:
> Here is background on the latest dataplane work in my "[PATCH v2 00/25]
> dataplane: use QEMU block layer" series.  It's necessary for anyone who wants
> to build on top of it.  Please leave feedback or questions and I'll submit a
> docs/ patch with the final version of this document.
> 
> 
> This document explains the IOThread feature and how to write code that runs
> outside the QEMU global mutex.
> 
> The main loop and IOThreads
> ---------------------------
> QEMU is an event-driven program that can do several things at once using an
> event loop.  The VNC server and the QMP monitor are both processed from the
> same event loop which monitors their file descriptors until they become
> readable and then invokes a callback.
> 
> The default event loop is called the main loop (see main-loop.c).  It is
> possible to create additional event loop threads using -object
> iothread,id=my-iothread.

Is dataplane the only user for this now?

> 
> Side note: The main loop and IOThread are both event loops but their code is
> not shared completely.  Sometimes it is useful to remember that although they
> are conceptually similar they are currently not interchangeable.
> 
> Why IOThreads are useful
> ------------------------
> IOThreads allow the user to control the placement of work.  The main loop is a
> scalability bottleneck on hosts with many CPUs.  Work can be spread across
> several IOThreads instead of just one main loop.  When set up correctly this
> can improve I/O latency and reduce jitter seen by the guest.
> 
> The main loop is also deeply associated with the QEMU global mutex, which is a
> scalability bottleneck in itself.  vCPU threads and the main loop use the QEMU
> global mutex to serialize execution of QEMU code.  This mutex is necessary
> because a lot of QEMU's code historically was not thread-safe.
> 
> The fact that all I/O processing is done in a single main loop and that the
> QEMU global mutex is contended by all vCPU threads and the main loop explain
> why it is desirable to place work into IOThreads.
> 
> The experimental virtio-blk data-plane implementation has been benchmarked and
> shows these effects:
> ftp://public.dhe.ibm.com/linux/pdfs/KVM_Virtualized_IO_Performance_Paper.pdf
> 
> How to program for IOThreads
> ----------------------------
> The main difference between legacy code and new code that can run in an
> IOThread is dealing explicitly with the event loop object, AioContext
> (see include/block/aio.h).  Code that only works in the main loop
> implicitly uses the main loop's AioContext.  Code that supports running
> in IOThreads must be aware of its AioContext.
> 
> AioContext supports the following services:
>  * File descriptor monitoring (read/write/error)
>  * Event notifiers (inter-thread signalling)
>  * Timers
>  * Bottom Halves (BH) deferred callbacks
> 
> There are several old APIs that use the main loop AioContext:
>  * LEGACY qemu_aio_set_fd_handler() - monitor a file descriptor
>  * LEGACY qemu_aio_set_event_notifier() - monitor an event notifier
>  * LEGACY timer_new_ms() - create a timer
>  * LEGACY qemu_bh_new() - create a BH
>  * LEGACY qemu_aio_wait() - run an event loop iteration
> 
> Since they implicitly work on the main loop they cannot be used in code that
> runs in an IOThread.  They might cause a crash or deadlock if called from an
> IOThread since the QEMU global mutex is not held.
> 
> Instead, use the AioContext functions directly (see include/block/aio.h):
>  * aio_set_fd_handler() - monitor a file descriptor
>  * aio_set_event_notifier() - monitor an event notifier
>  * aio_timer_new() - create a timer
>  * aio_bh_new() - create a BH
>  * aio_poll() - run an event loop iteration
> 
> The AioContext can be obtained from the IOThread using
> iothread_get_aio_context() or for the main loop using qemu_get_aio_context().
> This way your code works both in IOThreads or the main loop.

I think such code knows about its iothread, so iothread_get_aio_context is
enough, why need to mention (using) qemu_get_aio_context here?

> 
> How to synchronize with an IOThread
> -----------------------------------
> AioContext is not thread-safe so some rules must be followed when using file
> descriptors, event notifiers, timers, or BHs across threads:
> 
> 1. AioContext functions can be called safely from file descriptor, event
> notifier, timer, or BH callbacks invoked by the AioContext.  No locking is
> necessary.
> 
> 2. Other threads wishing to access the AioContext must use
> aio_context_acquire()/aio_context_release() for mutual exclusion.  Once the
> context is acquired no other thread can access it or run event loop iterations
> in this AioContext.
> 
> aio_context_acquire()/aio_context_release() calls may be nested.  This
> means you can call them if you're not sure whether #1 applies.
> 
> Side note: the best way to schedule a function call across threads is to 
> create
> a BH in the target AioContext beforehand and then call qemu_bh_schedule().  No
> acquire/release or locking is needed for the qemu_bh_schedule() call.  But be
> sure to acquire the AioContext for aio_bh_new() if necessary.
> 
> The relationship between AioContext and the block layer
> -------------------------------------------------------
> The AioContext originates from the QEMU block layer because it provides a
> scoped way of running event loop iterations until all work is done.  This
> feature is used to complete all in-flight block I/O requests (see
> bdrv_drain_all()).  Nowadays AioContext is a generic event loop that can be
> used by any QEMU subsystem.

There was a concern about lock ordering, currently we only acquire contexts
from main loop and vCPU threads, so we're safe. Do we enforce this rule? If we
use this reentrant lock in others parts of QEMU, what are the rules then?

> 
> The block layer has support for AioContext integrated.  Each BlockDriverState
> is associated with an AioContext using bdrv_set_aio_context() and
> bdrv_get_aio_context().  This allows block layer code to process I/O inside 
> the
> right AioContext.  Other subsystems may wish to follow a similar approach.
> 
> If main loop code such as a QMP function wishes to access a BlockDriverState 
> it
> must first call aio_context_acquire(bdrv_get_aio_context(bs)) to ensure the
> IOThread does not run in parallel.

Does it imply that adding aio_context_acquire and aio_context_release
protection inside a bdrv_* function makes it (at least in the sense of
IOThreads) thread-safe?

> 
> Long-running jobs (usually in the form of coroutines) are best scheduled in 
> the
> BlockDriverState's AioContext to avoid the need to acquire/release around each
> bdrv_*() call.  Be aware that there is currently no mechanism to get notified
> when bdrv_set_aio_context() moves this BlockDriverState to a different
> AioContext (see bdrv_detach_aio_context()/bdrv_attach_aio_context()), so you
> may need to add this if you want to support long-running jobs.

Is block job a case of this? Looks like a subtask of adding support of block
jobs in dataplane.

Thanks,
Fam