Re: [Qemu-devel] Linux multiqueue block layer thoughts

[Jens Axboe]

On Wed, Nov 27 2013, Stefan Hajnoczi wrote:
> I finally got around to reading the Linux multiqueue block layer paper
> and wanted to share some thoughts about how it relates to QEMU and
> dataplane/QContext:
> http://kernel.dk/blk-mq.pdf
> 
> I think Jens has virtio-blk multiqueue patches.  So let's imagine that
> the virtio-blk device has multiple virtqueues.  (virtio-scsi is
> already multiqueue BTW.)
> 
> The paper focusses on two queue mappings: 1 queue per core and 1 queue
> per node.  In both cases the idea is to keep the block I/O code path
> localized.  This makes block I/O scale as the number of CPUs
> increases.
> 
> In QEMU we'd want to set up a mapping for the virtio-blk mq device:
> each guest vcpu or guest node has a virtio-blk virtqueue which is
> serviced by a dataplane/QContext thread.
> 
> QEMU would then process requests across these queues in parallel,
> although currently BlockDriverState is not thread-safe.  At least for
> raw we should be able to submit requests in parallel from QEMU.
> 
> Unfortunately there are some complications in the QEMU block layer:
> QEMU's own accounting, request tracking, and throttling features are
> global.  We'd need to eventually do something similar to the
> multiqueue block layer changes in the kernel to detangle this state.
> 
> Doing multiqueue for image formats is much more challenging - we'd
> have to tackle thread-safety in qcow2 and friends.  For network block
> drivers like Gluster or NBD it's also not 100% clear what the best
> approach is.  But I think the target here is local SSDs that are
> capable of high IOPs together with an SMP guest.
> 
> At the end of all this we'd arrive at the following architecture:
> 1. Guest virtio device has multiple queues (1 per node or vcpu).
> 2. QEMU has multiple dataplane/QContext threads that process virtqueue
> kicks, they are bound to host CPUs/nodes.
> 3. Linux kernel has multiqueue block I/O.

I think that sounds very reasonable. Let me know if there's anything you
need help or advice with.

> Jens: when experimenting with multiqueue virtio-blk, how far did you
> modify QEMU to eliminate global request processing state from block.c?

I did very little scaling testing on virtio-blk, it was more a demo case
for conversion than anything else. So probably not of much use to what
you are looking for...

-- 
Jens Axboe