[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [Qemu-devel] [RFC PATCH V4 3/4] vfio: Add SaveVMHanlders for VFIO de
From: |
Zhang, Yulei |
Subject: |
Re: [Qemu-devel] [RFC PATCH V4 3/4] vfio: Add SaveVMHanlders for VFIO device to support live migration |
Date: |
Wed, 18 Apr 2018 10:57:50 +0000 |
> -----Original Message-----
> From: Alex Williamson [mailto:address@hidden
> Sent: Tuesday, April 17, 2018 10:35 PM
> To: Zhang, Yulei <address@hidden>
> Cc: address@hidden; Tian, Kevin <address@hidden>;
> address@hidden; address@hidden;
> address@hidden; Wang, Zhi A <address@hidden>;
> address@hidden; address@hidden
> Subject: Re: [RFC PATCH V4 3/4] vfio: Add SaveVMHanlders for VFIO device
> to support live migration
>
> On Tue, 17 Apr 2018 08:11:12 +0000
> "Zhang, Yulei" <address@hidden> wrote:
>
> > > -----Original Message-----
> > > From: Alex Williamson [mailto:address@hidden
> > > Sent: Tuesday, April 17, 2018 4:38 AM
> > > To: Zhang, Yulei <address@hidden>
> > > Cc: address@hidden; Tian, Kevin <address@hidden>;
> > > address@hidden; address@hidden;
> > > address@hidden; Wang, Zhi A <address@hidden>;
> > > address@hidden; address@hidden
> > > Subject: Re: [RFC PATCH V4 3/4] vfio: Add SaveVMHanlders for VFIO
> device
> > > to support live migration
> > >
> > > On Tue, 10 Apr 2018 14:03:13 +0800
> > > Yulei Zhang <address@hidden> wrote:
> > >
> > > > Instead of using vm state description, add SaveVMHandlers for VFIO
> > > > device to support live migration.
> > > >
> > > > Introduce new Ioctl VFIO_DEVICE_GET_DIRTY_BITMAP to fetch the
> > > memory
> > > > bitmap that dirtied by vfio device during the iterative precopy stage
> > > > to shorten the system downtime afterward.
> > > >
> > > > For vfio pci device status migrate, during the system downtime, it
> > > > will save the following states 1. pci configuration space addr0~addr5
> > > > 2. pci configuration space msi_addr msi_data 3. pci device status
> > > > fetch from device driver
> > > >
> > > > And on the target side the vfio_load will restore the same states 1.
> > > > re-setup the pci bar configuration 2. re-setup the pci device msi
> > > > configuration 3. restore the pci device status
> > >
> > > Interrupts are configured via ioctl, but I don't see any code here to
> configure
> > > the device into the correct interrupt state. How do we know the target
> > > device is compatible with the source device? Do we rely on the vendor
> > > driver to implicitly include some kind of device and version information
> and
> > > fail at the very end of the migration? It seems like we need to somehow
> > > front-load that sort of device compatibility checking since a vfio-pci
> device
> > > can be anything (ex. what happens if a user tries to migrate a GVT-g
> vGPU to
> > > an NVIDIA vGPU?). Thanks,
> > >
> > > Alex
> >
> > We emulate the write to the pci configure space in vfio_load() which will
> help
> > setup the interrupt state.
>
> But you're only doing that for MSI, not MSI-X, we cannot simply say
> that we don't have an MSI-X devices right now and add it later or we'll
> end up with incompatible vmstate, we need to plan for how we'll support
> it within the save state stream now.
>
Agree with u.
> > For compatibility I think currently the vendor driver would put version
> number
> > or device specific info in the device state region, so during the pre-copy
> stage
> > the target side will discover the difference and call off the migration.
>
> Those sorts of things should be built into the device state region, we
> shouldn't rely on vendor drivers to make these kinds of considerations,
> we should build it into the API, which also allows QEMU to check state
> compatibility before attempting a migration. Thanks,
>
> Alex
Not sure about how to check the compatibility before attempting a migration,
to my understanding, qemu doesn't know the configuration on target side,
target may call off the migration when it finds the input device state isn't
compatible,
and source vm will resume.
Re: [Qemu-devel] [RFC PATCH V4 3/4] vfio: Add SaveVMHanlders for VFIO device to support live migration, Dr. David Alan Gilbert, 2018/04/17