Re: [Qemu-devel] [RFC PATCH v1 1/1] vGPU core driver : to provide common interface for vGPU.

[Tian, Kevin]

> From: Kirti Wankhede [mailto:address@hidden
> Sent: Tuesday, February 02, 2016 9:48 AM
> 
> Resending this mail again, somehow my previous mail didn't reached every
> to everyone's inbox.
> 
> On 2/2/2016 3:16 AM, Kirti Wankhede wrote:
> > Design for vGPU Driver:
> > Main purpose of vGPU driver is to provide a common interface for vGPU
> > management that can be used by differnt GPU drivers.

Thanks for composing this design which is a good start.

> >
> > This module would provide a generic interface to create the device, add
> > it to vGPU bus, add device to IOMMU group and then add it to vfio group.
> >
> > High Level block diagram:
> >
> >
> > +--------------+    vgpu_register_driver()+---------------+
> > |     __init() +------------------------->+               |
> > |              |                          |               |
> > |              +<-------------------------+    vgpu.ko    |
> > | vfio_vgpu.ko |   probe()/remove()       |               |
> > |              |                +---------+               +---------+
> > +--------------+                |         +-------+-------+         |
> >                                  |                 ^                 |
> >                                  | callback        |                 |
> >                                  |         +-------+--------+        |
> >                                  |         |vgpu_register_device()   |
> >                                  |         |                |        |
> >                                  +---^-----+-----+    +-----+------+-+
> >                                      | nvidia.ko |    |  i915.ko   |
> >                                      |           |    |            |
> >                                      +-----------+    +------------+
> >
> > vGPU driver provides two types of registration interfaces:

Looks you missed callbacks which vgpu.ko provides to vfio_vgpu.ko,
e.g. to retrieve basic region/resource info, etc...

Also for GPU driver interfaces, better to identify the caller. E.g. it's
easy to understand life-cycle management would come from sysfs
by mgmt. stack like libvirt. What about @read and @write? what's
the connection between this vgpu core driver and specific hypervisor?
etc. Better to connect all necessary dots so we can refine all 
necessary requirements on this proposal.

[...]
> >
> > 2. GPU driver interface
> >
> > /**
> >   * struct gpu_device_ops - Structure to be registered for each physical
> > GPU to
> >   * register the device to vgpu module.
> >   *
> >   * @owner:              The module owner.
> >   * @vgpu_supported_config: Called to get information about supported
> >   *                       vgpu types.
> >   *                      @dev : pci device structure of physical GPU.
> >   *                      @config: should return string listing supported
> >   *                      config
> >   *                      Returns integer: success (0) or error (< 0)
> >   * @vgpu_create:        Called to allocate basic resouces in graphics
> >   *                      driver for a particular vgpu.
> >   *                      @dev: physical pci device structure on which vgpu
> >   *                            should be created
> >   *                      @uuid: uuid for which VM it is intended to
> >   *                      @instance: vgpu instance in that VM
> >   *                      @vgpu_id: This represents the type of vgpu to be
> >   *                                created
> >   *                      Returns integer: success (0) or error (< 0)

Specifically for Intel GVT-g we didn't hard partition resource among vGPUs.
Instead we allow user to accurately control how many physical resources
are allocated to a vGPU. So this interface should be extensible to allow
vendor specific resource control.

And for UUID, I remember Alex had a concern on using it in kernel. 
Honestly speaking I don't have a good idea here. In Xen side there is a VM ID
which can be easily used as the index. But for KVM, what would be the best
identifier to associate with a VM?

> >   * @vgpu_destroy:       Called to free resources in graphics driver for
> >   *                      a vgpu instance of that VM.
> >   *                      @dev: physical pci device structure to which
> >   *                      this vgpu points to.
> >   *                      @uuid: uuid for which the vgpu belongs to.
> >   *                      @instance: vgpu instance in that VM
> >   *                      Returns integer: success (0) or error (< 0)
> >   *                      If VM is running and vgpu_destroy is called that
> >   *                      means the vGPU is being hotunpluged. Return error
> >   *                      if VM is running and graphics driver doesn't
> >   *                      support vgpu hotplug.
> >   * @vgpu_start:         Called to do initiate vGPU initialization
> >   *                      process in graphics driver when VM boots before
> >   *                      qemu starts.
> >   *                      @uuid: UUID which is booting.
> >   *                      Returns integer: success (0) or error (< 0)
> >   * @vgpu_shutdown:      Called to teardown vGPU related resources for
> >   *                      the VM
> >   *                      @uuid: UUID which is shutting down .
> >   *                      Returns integer: success (0) or error (< 0)
> >   * @read:               Read emulation callback
> >   *                      @vdev: vgpu device structure
> >   *                      @buf: read buffer
> >   *                      @count: number bytes to read
> >   *                      @address_space: specifies for which address space

and suppose there'll be an 'offset' as required by usual emulation.

> >   *                      the request is: pci_config_space, IO register
> >   *                      space or MMIO space.
> >   *                      Retuns number on bytes read on success or error.
> >   * @write:              Write emulation callback
> >   *                      @vdev: vgpu device structure
> >   *                      @buf: write buffer
> >   *                      @count: number bytes to be written
> >   *                      @address_space: specifies for which address space
> >   *                      the request is: pci_config_space, IO register
> >   *                      space or MMIO space.
> >   *                      Retuns number on bytes written on success or
> > error.
> >   * @vgpu_set_irqs:      Called to send about interrupts configuration
> >   *                      information that qemu set.
> >   *                      @vdev: vgpu device structure
> >   *                      @flags, index, start, count and *data : same as
> >   *                      that of struct vfio_irq_set of
> >   *                      VFIO_DEVICE_SET_IRQS API.

any elaboration how this will be used in your case?

> >   *
> >   * Physical GPU that support vGPU should be register with vgpu module with
> >   * gpu_device_ops structure.
> >   */
> >

Also it'd be good design to allow extensible usages, such as statistics, and
other vendor specific control knobs (e.g. foreground/background VM switch
in Intel GVT-g, etc.)


Thanks
Kevin