[Qemu-devel] [RFC v1] Introduce a new NVMe host device type to QEMU

[Changpeng Liu]

NVMe 1.3 specification(http://nvmexpress.org/resources/specifications/) 
introduced a new Admin command:
Doorbell Buffer Config, which designed for emulated NVMe controllers only, 
Linux kernel 4.12 added the
support of Doorbell Buffer Config. With this feature, when NVMe driver issues 
new requests to firmware,
the driver will write shadow doorbell instead of MMIO writes, so the NVMe 
specification itself can
become a great Para-virtualization protocol.

While here, similar with existing vhost-user-scsi idea, we can setup a slave 
I/O target which can serve
Guest I/Os directly via NVMe I/O queues. Here we can route the NVMe queue's 
information, such as queue
size/queue address etc. to a separate slave I/O target via UNIX domain socket. 
I took exist QEMU
vhost-user protocol as reference, designed several totally new socket messages 
to enable the function.
With this idea, an emulated virtual NVMe controller  will be presented at the 
Guest, and native NVMe
driver inside Guest can be used.

-----------------------------------------------------------------------------------------------------------------------------------------
| Unix Domain Socket Messages      | Description                                
                                                                | 
-----------------------------------------------------------------------------------------------------------------------------------------
| Get Controller Capabilities             | Controller capabilitiy register of 
NVMe specification                        |
-----------------------------------------------------------------------------------------------------------------------------------------
| Get/Set Controller Configuration | Enable/Disable NVMe controller             
                                               |
-----------------------------------------------------------------------------------------------------------------------------------------
| Admin passthrough                        | Mandatory NVMe Admin commands 
routed to slave I/O target      |
-----------------------------------------------------------------------------------------------------------------------------------------
| IO passthrough                               | IO messages before the shadow 
doorbell buffer being configured  |
-----------------------------------------------------------------------------------------------------------------------------------------
| Set memory table                          | Same with exist vhost-user 
message, used for memory translation |
-----------------------------------------------------------------------------------------------------------------------------------------
| Set Guest Notifier                          | Completion queue interrupt, 
interrupt Guest when I/O completed |
-----------------------------------------------------------------------------------------------------------------------------------------

With those messages, slave I/O target can access all the I/O queues of NVMe 
include submission queue and
completion queue. After finished the Admin Shadow Doorbell command, the slave 
I/O target can start to
process the I/O requests sent from Guest.

Currently I implemented both QEMU driver and slave I/O target which largely 
reused the code from QEMU
NVMe driver and vhost-user driver for performance evaluation:

Optional slave I/O target(SPDK Vhost Target) patches: 
https://review.gerrithub.io/#/c/384213/

User space NVMe driver is implemented at the slave I/O target so that NVMe 
controller can be shared
with multiple VMs, and the namespaces presented to the guest VM are virtual 
namespaces, meaning the
slave I/O target can back these namespaces with any kind of storage. Guest OS 
must be 4.12 or later(with
Admin Doorbell Buffer Config support), tests from my side used Fedora 27 with 
4.13 kernel.

Currently this still is an ongoing work, there are some opens need to be 
addressed:
-Reused a lot of code from QEMU/nvme driver, need to think about abstracting a 
common NVMe library;
-Reused a lot of code from QEMU/vhost-user driver, for this idea, we just want 
to use UNIX domain
 socket to deliver mandatory messages, of course Set memory table and Set guest 
notifier is exactly
 same with vhost-user driver;
-Can support Guest OS kernel > 4.12 with Admin Doorbell Buffer feature enabled 
inside Guest, for BIOS
 stage IO requests and older Linux kernel without Admin Doorbell Buffer 
support, it can forward the IO
 requests through socket message, but this will have huge performance drop;

Any feedback is appreciated.

Changpeng Liu (1):
  block/NVMe: introduce a new vhost NVMe host device to QEMU

 hw/block/Makefile.objs     |   3 +
 hw/block/nvme.h            |  28 ++
 hw/block/vhost.c           | 439 ++++++++++++++++++++++
 hw/block/vhost_user.c      | 588 +++++++++++++++++++++++++++++
 hw/block/vhost_user_nvme.c | 902 +++++++++++++++++++++++++++++++++++++++++++++
 hw/block/vhost_user_nvme.h |  38 ++
 6 files changed, 1998 insertions(+)
 create mode 100644 hw/block/vhost.c
 create mode 100644 hw/block/vhost_user.c
 create mode 100644 hw/block/vhost_user_nvme.c
 create mode 100644 hw/block/vhost_user_nvme.h

-- 
1.9.3