Enable vfio-pci devices to be saved and restored across a cpr-exec of qemu.
At vfio creation time, save the value of vfio container, group, and device
descriptors in CPR state.
In the container pre_save handler, suspend the use of virtual addresses
in DMA mappings with VFIO_DMA_UNMAP_FLAG_VADDR, because guest ram will
be remapped at a different VA after exec. DMA to already-mapped pages
continues. Save the msi message area as part of vfio-pci vmstate, and
save the interrupt and notifier eventfd's in vmstate.
On qemu restart, vfio_realize() finds the saved descriptors, uses the
descriptors, and notes that the device is being reused. Device and iommu
state is already configured, so operations in vfio_realize that would
modify the configuration are skipped for a reused device, including vfio
ioctl's and writes to PCI configuration space. Vfio PCI device reset
is also suppressed. The result is that vfio_realize constructs qemu
data structures that reflect the current state of the device. However,
the reconstruction is not complete until migrate_incoming is called.
migrate_incoming loads the msi data, the vfio post_load handler finds
eventfds in CPR state, rebuilds vector data structures, and attaches the
interrupts to the new KVM instance. The container post_load handler then
invokes the main vfio listener callback, which walks the flattened ranges
of the vfio address space and calls VFIO_DMA_MAP_FLAG_VADDR to inform the
kernel of the new VA's. Lastly, migration resumes the VM.