Re: [Qemu-devel] [RFC 5/5] vifo: introduce new VFIO ioctl VFIO_DEVICE_PCI_GET_DIRTY_BITMAP

[Tian, Kevin]

> From: Alex Williamson [mailto:address@hidden
> Sent: Friday, June 30, 2017 4:57 AM
> 
> On Thu, 29 Jun 2017 00:10:59 +0000
> "Tian, Kevin" <address@hidden> wrote:
> 
> > > From: Alex Williamson [mailto:address@hidden
> > > Sent: Thursday, June 29, 2017 12:00 AM
> > > Thanks Kevin.  So really it's not really a dirty bitmap, it's just a
> > > bitmap of pages that the device has access to and may have dirtied.
> > > Don't we have this more generally in the vfio type1 IOMMU backend?  For
> > > a mediated device, we know all the pages that the vendor driver has
> > > asked to be pinned.  Should we perhaps make this interface on the vfio
> > > container rather than the device?  Any mediated device can provide this
> > > level of detail without specific vendor support.  If we had DMA page
> > > faulting, this would be the natural place to put it as well, so maybe
> > > we should design the interface there to support everything similarly.
> > > Thanks,
> > >
> >
> > That's a nice idea. Just two comments:
> >
> > 1) If some mediated device has its own way to construct true dirty
> > bitmap (not thru DMA page faulting), the interface is better designed
> > to allow that flexibility. Maybe an optional callback if not registered
> > then use common type1 IOMMU logic otherwise prefers to vendor
> > specific callback
> 
> I'm not sure what that looks like, but I agree with the idea.  Could
> the pages that type1 knows about every be anything other than a
> superset of the dirty pages?  Perhaps a device ioctl to flush unused
> mappings would be sufficient.

sorry I didn't quite get your idea here. My understanding is that
type1 is OK as an alternative in case mediated device has no way
to track dirtied pages (as for Intel GPU), so we can use type1 pinned
pages as an indirect way to indicate dirtied pages. But if mediated
device has its own way (e.g. a device private MMU) to track dirty
pages, then we should allow that device to provide dirty bitmap
instead of using type1.

> 
> > 2) If there could be multiple mediated devices from different vendors
> > in same container while not all mediated devices support live migration,
> > would container-level interface impose some limitation?
> 
> Dirty page logging is only one small part of migration, each
> migrate-able device would still need to provide a device-level
> interface to save/restore state.  The migration would fail when we get
> to the device(s) that don't provide that.  Thanks,
> 

Agree here. Yulei, can you investigate this direction and report back
whether it's feasible or anything overlooked?

Thanks
Kevin