Re: [Qemu-devel] [RFC PATCH v1 1/1] vGPU core driver : to provide common interface for vGPU.

[Tian, Kevin]

> From: Kirti Wankhede
> Sent: Tuesday, February 02, 2016 5:25 PM
> 
> On 2/2/2016 1:12 PM, Tian, Kevin wrote:
> >> From: Kirti Wankhede [mailto:address@hidden
> >> Sent: Tuesday, February 02, 2016 9:48 AM
> >>
> >> Resending this mail again, somehow my previous mail didn't reached every
> >> to everyone's inbox.
> >>
> >> On 2/2/2016 3:16 AM, Kirti Wankhede wrote:
> >>> Design for vGPU Driver:
> >>> Main purpose of vGPU driver is to provide a common interface for vGPU
> >>> management that can be used by differnt GPU drivers.
> >
> > Thanks for composing this design which is a good start.
> >
> >>>
> >>> This module would provide a generic interface to create the device, add
> >>> it to vGPU bus, add device to IOMMU group and then add it to vfio group.
> >>>
> >>> High Level block diagram:
> >>>
> >>>
> >>> +--------------+    vgpu_register_driver()+---------------+
> >>> |     __init() +------------------------->+               |
> >>> |              |                          |               |
> >>> |              +<-------------------------+    vgpu.ko    |
> >>> | vfio_vgpu.ko |   probe()/remove()       |               |
> >>> |              |                +---------+               +---------+
> >>> +--------------+                |         +-------+-------+         |
> >>>                                   |                 ^                 |
> >>>                                   | callback        |                 |
> >>>                                   |         +-------+--------+        |
> >>>                                   |         |vgpu_register_device()   |
> >>>                                   |         |                |        |
> >>>                                   +---^-----+-----+    +-----+------+-+
> >>>                                       | nvidia.ko |    |  i915.ko   |
> >>>                                       |           |    |            |
> >>>                                       +-----------+    +------------+
> >>>
> >>> vGPU driver provides two types of registration interfaces:
> >
> > Looks you missed callbacks which vgpu.ko provides to vfio_vgpu.ko,
> > e.g. to retrieve basic region/resource info, etc...
> >
> 
> Basic region info or resource info would come from GPU driver. So
> retrieving such info should be the part of GPU driver interface. Like I
> mentioned we need to enhance these interfaces during development as and
> when we find it useful.
> vfio_vgpu.ko gets dev pointer from which it can reach to vgpu_device
> structure and then it can use GPU driver interface directly to retrieve
> such information from GPU driver.

Thanks for confirmation. Want to make sure we are on the same page
since those callbacks are not listed in current proposal yet.

> 
> This RFC is to focus more on different modules and its structures, how
> those modules would be inter-linked with each other and have a flexible
> design to keep the scope for enhancements.
> 
> We have identified three modules:
> 
> * vgpu.ko - vGPU core driver that provide registration interface for GPU
> driver and vGPU VFIO  driver, responsible for creating vGPU devices and
> providing management interface for vGPU devices.
> * vfio_vgpu.ko - vGPU VFIO driver for vGPU device, provides VFIO
> interface that is used by QEMU.
> * vfio_iommu_type1_vgpu.ko - IOMMU TYPE1 driver supporting the IOMMU
> TYPE1 v1 and v2 interface.
> 
> The above block diagram gives an overview how vgpu.ko, vfio_vgpu.ko and
> GPU drivers would be inter-linked with each other.

This part is clear.

> >>>    * @vgpu_create:        Called to allocate basic resouces in graphics
> >>>    *                      driver for a particular vgpu.
> >>>    *                      @dev: physical pci device structure on which 
> >>> vgpu
> >>>    *                            should be created
> >>>    *                      @uuid: uuid for which VM it is intended to
> >>>    *                      @instance: vgpu instance in that VM
> >>>    *                      @vgpu_id: This represents the type of vgpu to be
> >>>    *                                created
> >>>    *                      Returns integer: success (0) or error (< 0)
> >
> > Specifically for Intel GVT-g we didn't hard partition resource among vGPUs.
> > Instead we allow user to accurately control how many physical resources
> > are allocated to a vGPU. So this interface should be extensible to allow
> > vendor specific resource control.
> >
> 
> This interface forwards the create request to vendor/GPU driver
> informing about which physical GPU this request is intended for and the
> type of vGPU. Then its vendor/GPU driver's responsibility to do
> resources allocation and manage resources in their own driver.

However the current parameter definition disallows resource configuration
information passed from user. As I said, Intel GVT-g doesn't do static
allocation based on type. We provide flexibility to user for fine-grained
resource management.

> >>>    *
> >>>    * Physical GPU that support vGPU should be register with vgpu module 
> >>> with
> >>>    * gpu_device_ops structure.
> >>>    */
> >>>
> >
> > Also it'd be good design to allow extensible usages, such as statistics, and
> > other vendor specific control knobs (e.g. foreground/background VM switch
> > in Intel GVT-g, etc.)
> >
> 
> Can you elaborate on what other control knobs that would be needed?
> 

Some examples:

- foreground/background VM switch
- resource query
- various statistics info
- virtual monitor configuration
- ...

Since this vgpu core driver will become the central point for all vgpu
management, we need provide an easy way for vendor specific extension,
e.g. exposing above callbacks as sysfs nodes and then vendor can create
its own extensions under subdirectory (/intel, /nvidia, ...).


Thanks
Kevin