Re: [Qemu-devel] [PATCH V3 0/4] vfio: Introduce Live migration capability to vfio_mdev device

[Zhang, Yulei]

> -----Original Message-----
> From: Alex Williamson [mailto:address@hidden
> Sent: Tuesday, March 13, 2018 6:22 AM
> To: Zhang, Yulei <address@hidden>
> Cc: address@hidden; Tian, Kevin <address@hidden>;
> address@hidden; address@hidden; Juan Quintela
> <address@hidden>
> Subject: Re: [PATCH V3 0/4] vfio: Introduce Live migration capability to
> vfio_mdev device
> 
> [cc +Juan]
> 
> On Mon,  5 Mar 2018 14:00:49 +0800
> Yulei Zhang <address@hidden> wrote:
> 
> > Summary
> >
> > This series RFC would like to resume the discussion about how to
> > introduce the live migration capability to vfio mdev device.
> >
> > By adding a new vfio subtype region
> VFIO_REGION_SUBTYPE_DEVICE_STATE,
> > the mdev device will be set to migratable if the new region exist
> > during the initialization.
> >
> > The intention to add the new region is using it for mdev device status
> > save and restore during the migration. The access to this region
> > will be trapped and forward to the mdev device driver, it also uses
> > the first byte in the new region to control the running state of mdev
> > device, so during the migration after stop the mdev driver, qemu could
> > retrieve the specific device status from this region and transfer to
> > the target VM side for the mdev device restore.
> >
> > In addition,  we add one new ioctl VFIO_IOMMU_GET_DIRTY_BITMAP to
> help do
> > the mdev device dirty page synchronization during the migration,
> currently
> > it is just for static copy, in the future we would like to add new interface
> > for the pre-copy.
> 
> Juan had concerns about another dirty bitmap implementation.  I'm not
> sure what alternatives we have, but let's loop him in for guidance on
> the best migration strategies.  The migration state for a device could
> be many gigabytes.
> 
> > Below is the vfio_mdev device migration sequence
> > Source VM side:
> >                     start migration
> >                             |
> >                             V
> >              get the cpu state change callback, write to the
> >              subregion's first byte to stop the mdev device
> >                             |
> >                             V
> >              quary the dirty page bitmap from iommu container
> >              and add into qemu dirty list for synchronization
> >                             |
> >                             V
> >              save the deivce status into Qemufile which is
> >                      read from the vfio device subregion
> >
> > Target VM side:
> >                restore the mdev device after get the
> >                  saved status context from Qemufile
> >                             |
> >                             V
> >                  get the cpu state change callback
> >                  write to subregion's first byte to
> >                       start the mdev device to put it in
> >                       running status
> >                             |
> >                             V
> >                     finish migration
> >
> > V3->V2:
> > 1. rebase the patch to Qemu stable 2.10 branch.
> > 2. use a common name for the subregion instead of specific for
> >    intel IGD.
> 
> But it's still tied to Intel's vendor ID??
> 
No. this is not necessary, I will remove the intel vendor ID. 

> Thanks,
> Alex
> 
> 
> >
> > V1->V2:
> > Per Alex's suggestion:
> > 1. use device subtype region instead of VFIO PCI fixed region.
> > 2. remove unnecessary ioctl, use the first byte of subregion to
> >    control the running state of mdev device.
> > 3. for dirty page synchronization, implement the interface with
> >    VFIOContainer instead of vfio pci device.
> >
> > Yulei Zhang (4):
> >   vfio: introduce a new VFIO subregion for mdev device migration support
> >   vfio: Add vm status change callback to stop/restart the mdev device
> >   vfio: Add struct vfio_vmstate_info to introduce put/get callback
> >     funtion for vfio device status save/restore
> >   vifo: introduce new VFIO ioctl VFIO_IOMMU_GET_DIRTY_BITMAP
> >
> >  hw/vfio/common.c              |  34 +++++++++
> >  hw/vfio/pci.c                 | 171
> +++++++++++++++++++++++++++++++++++++++++-
> >  hw/vfio/pci.h                 |   1 +
> >  include/hw/vfio/vfio-common.h |   1 +
> >  linux-headers/linux/vfio.h    |  29 ++++++-
> >  5 files changed, 232 insertions(+), 4 deletions(-)
> >