[Qemu-devel] [RFC Design Doc v2] Enable Shared Virtual Memory feature in pass-through scenarios

[Liu, Yi L]

What's changed from v1:
a) Solution changed: 
Switch to extend the existing QEMU vIOMMU emulator to support virtual SVM
for pass-through devices. Take use of VFIO to do data propagation.

b) A SVM Virtualization Architecture diagram is added.

Content
===============================================
1. Feature description
2. Why use it?
3. How to enable it
4. How to test

Details
===============================================
1. Feature description
Shared virtual memory(SVM) is to let application program share its virtual
address with SVM capable devices. And the feature in this design is to let
application programs(running in guest)share their virtual address with
assigned device(e.g. graphics processors or accelerators).

For SVM detail, you may want refer to section 2.5.1.1 of Intel VT-d spec and
section 5.6 of OpenCL spec. For details about SVM address translation,
pls refer to section 3 of Intel VT-d spec.
It's also welcomed to discuss directly in this thread.

Link to related specs:
http://www.intel.com/content/dam/www/public/us/en/documents/product-specifications/vt-directed-io-spec.pdf
https://www.khronos.org/registry/cl/specs/opencl-2.0.pdf


2. Why use it?
It is common to pass-through devices to guest and expect to achieve as
much similar performance as it is on host. With this feature enabled, 
the application programs in guest would be able to pass data-structures
to assigned devices without unnecessary overheads.


3. How to enable it
There is an existing IOMMU emulator in host user space(QEMU). The solution here
is to extend IOMMU emulator to support SVM for assigned devices. So far, the
vIOMMU exposed to guest is only for emulated devices. In this design, it would
focus on virtual SVM for assigned devices. Virtual IOVA and virtual interrupt
remapping will not be included here.

The enabling work would include the following items.

a) IOMMU Register Access Emulation
Already existed in QEMU, need some extensions to support SVM. e.g. support
page request service related registers.

b) vIOMMU Capability
Report SVM related capabilities(PASID,PRS,DT etc.) and cache mode in vIOMMU
capability/ex-capability register.

c) QI Handling Emulation
Already existed in QEMU, need to shadow the QIs related to assigned devices to
physical IOMMU.
i. ex-context entry cache invalidation(nested mode setting, guest PASID table
pointer shadowing)
ii. 1st level translation cache invalidation
iii. Response for recoverable faults

d) Address translation in virtual SVM
For requests with PASID from assigned device, the address translation would be
subjected to nested mode. For the SVM capable devices which are assigned to a
guest, the corresponding extended context entry should have the NESTE bit set.
Guest PASID table pointer would be shadowed to ex-context entry. It would be a
GPA as nested mode is on.

e) Recoverable Address Translation Faults Handling Emulation
It is serviced by page request when device support PRS. For assigned devices, 
host IOMMU driver would get page requests from pIOMMU. Here, we need a
mechanism to drain the page requests from devices which are assigned to a guest.
In this design it would be done through VFIO. Page request descriptors would be
propagated to user space and then exposed to guest IOMMU driver. This
requires following support:
i. a mechanism to notify vIOMMU emulator to fetch PRQ descriptor
ii. a notify framework in QEMU to signal the PRQ descriptor fetching when
notified

f) Non-Recoverable Address Translation Handling Emulation
The non-recoverable fault propagation is similar to recoverable faults. In this
design it would propagate fault data to user space(QEMU) through VFIO. 
vIOMMU emulator then emulate the fault. Either fill data to vIOMMU fault
record registers or fill the data to memory-resident fault log region. Depends
on the fault reporting type.

g) SVM Virtualization Architecture
**********************************************************************
 Guest                                 +------------------+           
                                    +->|   vIOMMU driver  |           
                                    |  +------------------+           
                                    |         |         |             
                                    +----+(1) |(2)      |(3)          
                                         |    |         |             
*****************************************|****|*********|*************
 Host User                               |    V         V             
 Space                                +------------------------+      
                                      |       Qemu vIOMMU      |      
                                      +------------------------+      
                                         |    |    |    |    |        
*****************************************|****|****|****|****|********
 Host Kernel                             |(1) |(2) |(4) |(5) |(6)     
 Space                                   |    |    |    |    |        
     +-----------------------+       +---------------------------+    
     |      IOMMU Fault      |       |   |    |    |    |    |   |    
     |  +--------------------|-------|---+    |    |    |    |   |    
     |  |                    |       |        |    |    |    |   |    
     |  |  +-----------------|-------|--------+    |    |    |   |    
     |  |  |                 |       |             |    |    |   |    
     |  |  |    IOMMU        |       |        VFIO |    |    |   |    
     |  |  |    Driver       |       |             |    |    |   |    
     |  |  |        +--------|-------|-------------+    |    |   |    
     |  |  |        |        |       |                  |    |   |    
     |  |  |        |  +-----|-------|------------------+    |   |    
     |  |  |        |  |     |       |                       |   |    
     |  |  |        |  |  +--|-------|-----------------------+   |    
     |  |  |        |  |  |  |       |                           |    
     +-----------------------+       +---------------------------+    
        |  |        |  |  |                                           
********|**|********|**|**|*******************************************
 HW     |  |        |  |  |                                           
        |  V        V  V  V                                           
      +----------------------+                                        
      |       pIOMMU         |                                        
      +----------------------+                                        
                                                                      
**********************************************************************
(1)Fault reporting, include recoverable and un-recoverable faults
(2)PRQ response
(3)Translation cache invalidation(QI)
(4)Set nested mode in pIOMMU ex-context entry
(5)Shadow gPASID table pointer to pIOMMU ex-context entry
(6)Cache invalidation for 1st level translation
<if the diagram is disordered, you may want to paste it to a Linux email client>

4. How to test
Test would be done with devices which has SVM capability. Hereby, Intel i915
GPU would be chosen to do the verification. Intel provides three tools from for
SVM verification. They are:
i)      intel-gpu-tools/tests/gem_svm_sanity
ii)     intel-gpu-tools/tests/gem_svm_fault
iii)    intel-gpu-tools/tests/gem_svm_storedw_loop_render

The following scenarios would have to be covered:

a) Test case 1 - SVM usage in host
i)      Requires a physical machine which has at least one SVM capable device. 
ii)     Run Test Tools in host. 
iii)    Expect: with vSVM enabled, it shouldn't affect SVM usage in host

b) Test case 2 - SVM usage in guest
i)      Requires a physical machine which has at least one SVM capable device. 
ii)     Create a guest, and assign a SVM capable device to it. 
iii)    Run Test Tools in the guest.
iV)     Expect: with vSVM enabled and device assigned, guest should be able to
use SVM with the assigned device

c) Test case 3 - SVM usage in multi-guests scenario
i)      Requires a physical machine which has at least two SVM capable devices. 
ii)     Create two guests, and assign a SVM capable device to each of them. 
iii)    Run Test Tools on both of the two guests.
iV)     Expect: multi-guest should be able to use SVM with its assigned
devices without affect each other

d) Test case 4 - SVM usage in host/guest scenario 
i)      Requires a physical machine which has at least two SVM capable devices. 
ii)     Create a guest, and assign a SVM capable device to the guest. 
iii)    Run Test Tools on both of the host and the guest.
iV)     Expect: host and guest shouldn't affect each other

Thanks,

Best Wishes,
Yi Liu