Re: [Qemu-devel] [RFC V2 0/4] vfio: Introduce Live migration capability to vfio_mdev device

[Zhang, Yulei]

> -----Original Message-----
> From: Tian, Kevin
> Sent: Monday, July 31, 2017 2:55 PM
> To: Zhang, Yulei <address@hidden>; address@hidden
> Cc: address@hidden; address@hidden; Zheng, Xiao
> <address@hidden>; Wang, Zhi A <address@hidden>;
> address@hidden
> Subject: RE: [Qemu-devel][RFC V2 0/4] vfio: Introduce Live migration
> capability to vfio_mdev device
> 
> > From: Zhang, Yulei
> > Sent: Tuesday, May 9, 2017 3:59 PM
> >
> > Summary
> >
> > This series RFC would like to introduce the live migration capability
> > to vfio_mdev device.
> >
> > As currently vfio_mdev device don't support migration, we introduce a
> > new vfio subtype region
> VFIO_REGION_SUBTYPE_INTEL_IGD_DEVICE_STATE
> > for Intel vGPU device, during the vfio device initialization, the mdev
> > device will be set to migratable if the new region exist.
> 
> Looking at your series, there is really nothing specific to vGPU or even Intel
> vGPU regarding to device state save/restore...
> 

You are right, we may define a new subtype for it as it is common region for 
vfio 
device state save/restore.

> >
> > The intention to add the new region is using it for vfio_mdev device
> > status save and restore during the migration. The access to this
> > region will be trapped and forward to the vfio_mdev device driver. And
> > we use the first byte in the new region to control the running state
> > of mdev device.
> >
> > Meanwhile we add one new ioctl VFIO_IOMMU_GET_DIRTY_BITMAP to
> help do
> > the mdev device dirty page synchronization.
> >
> > So the vfio_mdev device migration sequence would be Source VM side:
> >                     start migration
> >                             |
> >                             V
> >              get the cpu state change callback, write to the
> >              subregion's first byte to stop the mdev device
> >                             |
> >                             V
> >              quary the dirty page bitmap from iommu container
> >              and add into qemu dirty list for synchronization
> >                             |
> >                             V
> >              save the deivce status into Qemufile which is
> >                      read from the vfio device subregion
> >
> > Target VM side:
> >                restore the mdev device after get the
> >                  saved status context from Qemufile
> >                             |
> >                             V
> >                  get the cpu state change callback
> >                  write to subregion's first byte to
> >                       start the mdev device to put it in
> >                       running status
> >                             |
> >                             V
> >                     finish migration
> >
> > V1->V2:
> > Per Alex's suggestion:
> > 1. use device subtype region instead of VFIO PCI fixed region.
> > 2. remove unnecessary ioctl, use the first byte of subregion to
> >    control the running state of mdev device.
> > 3. for dirty page synchronization, implement the interface with
> >    VFIOContainer instead of vfio pci device.
> >
> > Yulei Zhang (4):
> >   vfio: introduce a new VFIO sub region for mdev device migration
> >     support
> >   vfio: Add vm status change callback to stop/restart the mdev device
> >   vfio: Add struct vfio_vmstate_info to introduce put/get callback
> >     funtion     for vfio device status save/restore
> >   vifo: introduce new VFIO ioctl VFIO_IOMMU_GET_DIRTY_BITMAP
> >
> >  hw/vfio/common.c              |  32 +++++++++
> >  hw/vfio/pci.c                 | 164
> > +++++++++++++++++++++++++++++++++++++++++-
> >  hw/vfio/pci.h                 |   1 +
> >  include/hw/vfio/vfio-common.h |   1 +
> >  linux-headers/linux/vfio.h    |  26 ++++++-
> >  5 files changed, 220 insertions(+), 4 deletions(-)
> >
> > --
> > 2.7.4