Re: [Qemu-devel] [PATCH 1/5] VFIO KABI for migration interface

[Dr. David Alan Gilbert]

* Kirti Wankhede (address@hidden) wrote:
> 
> 
> On 11/23/2018 12:24 AM, Dr. David Alan Gilbert wrote:
> > * Kirti Wankhede (address@hidden) wrote:
> >> - Defined MIGRATION region type and sub-type.
> >> - Defined VFIO device states during migration process.
> >> - Defined vfio_device_migration_info structure which will be placed at 0th
> >>   offset of migration region to get/set VFIO device related information.
> >>   Defined actions and members of structure usage for each action:
> >>     * To convey VFIO device state to be transitioned to.
> >>     * To get pending bytes yet to be migrated for VFIO device
> >>     * To ask driver to write data to migration region and return number of 
> >> bytes
> >>       written in the region
> >>     * In migration resume path, user space app writes to migration region 
> >> and
> >>       communicates it to vendor driver.
> >>     * Get bitmap of dirty pages from vendor driver from given start address
> >>
> >> Signed-off-by: Kirti Wankhede <address@hidden>
> >> Reviewed-by: Neo Jia <address@hidden>
> > 
> > <snip>
> > 
> >> + * Action Get buffer:
> >> + *      On this action, vendor driver should write data to migration 
> >> region and
> >> + *      return number of bytes written in the region.
> >> + *      data.offset [output] : offset in the region from where data is 
> >> written.
> >> + *      data.size [output] : number of bytes written in migration buffer 
> >> by
> >> + *          vendor driver.
> > 
> > <snip>
> > 
> >> + */
> >> +
> >> +struct vfio_device_migration_info {
> >> +        __u32 device_state;         /* VFIO device state */
> >> +        struct {
> >> +            __u64 precopy_only;
> >> +            __u64 compatible;
> >> +            __u64 postcopy_only;
> >> +            __u64 threshold_size;
> >> +        } pending;
> >> +        struct {
> >> +            __u64 offset;           /* offset */
> >> +            __u64 size;             /* size */
> >> +        } data;
> > 
> > I'm curious how the offsets/size work; how does the 
> > kernel driver know the maximum size of state it's allowed to write?
> 
> 
> Migration region looks like:
>  ----------------------------------------------------------------------
> |vfio_device_migration_info|    data section                        | 
> |                          |     ///////////////////////////////////  |
>  ----------------------------------------------------------------------
>  ^                             ^                                 ^
>  offset 0-trapped part         data.offset                     data.size
> 
> 
> Kernel driver defines the size of migration region and tells VFIO user
> space application (QEMU here) through VFIO_DEVICE_GET_REGION_INFO ioctl.
> So kernel driver can calculate the size of data section. Then kernel
> driver can have (data.size >= data section size) or (data.size < data
> section size), hence VFIO user space application need to know data.size
> to copy only relevant data.
> 
> > Why would it pick a none-0 offset into the output region?
> 
> Data section is always followed by vfio_device_migration_info structure
> in the region, so data.offset will always be none-0.
> Offset from where data is copied is decided by kernel driver, data
> section can be trapped or mapped depending on how kernel driver defines
> data section. If mmapped, then data.offset should be page aligned, where
> as initial section which contain vfio_device_migration_info structure
> might not end at offset which is page aligned.

Ah OK; I see - it wasn't clear to me which buffer we were talking about
here; so yes it makes sense if it's one the kernel had the control of.

Dave

> Thanks,
> Kirti
> 
> > Without having dug further these feel like i/o rather than just output;
> > i.e. the calling process says 'put it at that offset and you've got size
> > bytes' and the kernel replies with 'I did put it at offset and I wrote
> > only this size bytes'
> > 
> > Dave
> > 
> >> +        struct {
> >> +            __u64 start_addr;
> >> +            __u64 total;
> >> +            __u64 copied;
> >> +        } dirty_pfns;
> >> +} __attribute__((packed));
> >> +
> >>  /* -------- API for Type1 VFIO IOMMU -------- */
> >>  
> >>  /**
> >> -- 
> >> 2.7.0
> >>
> > --
> > Dr. David Alan Gilbert / address@hidden / Manchester, UK
> > 
--
Dr. David Alan Gilbert / address@hidden / Manchester, UK