Re: [Qemu-devel] [RFC 00/13] Live memory snapshot based on userfaultfd

[Baptiste Reynal]

On Thu, Jan 7, 2016 at 1:19 PM, zhanghailiang
<address@hidden> wrote:
> For now, we still didn't support live memory snapshot, we have discussed
> a scheme which based on userfaultfd long time ago.
> You can find the discussion by the follow link:
> https://lists.nongnu.org/archive/html/qemu-devel/2014-11/msg01779.html
>
> The scheme is based on userfaultfd's write-protect capability.
> The userfaultfd write protection feature is available here:
> http://www.spinics.net/lists/linux-mm/msg97422.html
>
> The process of this live memory scheme is like bellow:
> 1. Pause VM
> 2. Enable write-protect fault notification by using userfaultfd to
>    mark VM's memory to write-protect (readonly).
> 3. Save VM's static state (here is device state) to snapshot file
> 4. Resume VM, VM is going to run.
> 5. Snapshot thread begins to save VM's live state (here is RAM) into
>    snapshot file.
> 6. During this time, all the actions of writing VM's memory will be blocked
>   by kernel, and kernel will wakeup the fault treating thread in qemu to
>   process this write-protect fault. The fault treating thread will deliver 
> this
>   page's address to snapshot thread.
> 7. snapshot thread gets this address, save this page into snasphot file,
>    and then remove the write-protect by using userfaultfd API, after that,
>    the actions of writing will be recovered.
> 8. Repeat step 5~7 until all VM's memory is saved to snapshot file
>
> Compared with the feature of 'migrate VM's state to file',
> the main difference for live memory snapshot is it has little time delay for
> catching VM's state. It just captures the VM's state while got users snapshot
> command, just like take a photo of VM's state.
>
> For now, we only support tcg accelerator, since userfaultfd is not supporting
> tracking write faults for KVM.
>
> Usage:
> 1. Take a snapshot
> #x86_64-softmmu/qemu-system-x86_64 -machine pc-i440fx-2.5,accel=tcg,usb=off 
> -drive 
> file=/mnt/windows/win7_install.qcow2.bak,if=none,id=drive-ide0-0-1,format=qcow2,cache=none
>  -device ide-hd,bus=ide.0,unit=1,drive=drive-ide0-0-1,id=ide0-0-1  -vnc :7 -m 
> 8192 -smp 1 -netdev tap,id=bn0 -device virtio-net-pci,id=net-pci0,netdev=bn0  
> --monitor stdio
> Issue snapshot command:
> (qemu)migrate -d file:/home/Snapshot
> 2. Revert to the snapshot
> #x86_64-softmmu/qemu-system-x86_64 -machine pc-i440fx-2.5,accel=tcg,usb=off 
> -drive 
> file=/mnt/windows/win7_install.qcow2.bak,if=none,id=drive-ide0-0-1,format=qcow2,cache=none
>  -device ide-hd,bus=ide.0,unit=1,drive=drive-ide0-0-1,id=ide0-0-1  -vnc :7 -m 
> 8192 -smp 1 -netdev tap,id=bn0 -device virtio-net-pci,id=net-pci0,netdev=bn0  
> --monitor stdio -incoming file:/home/Snapshot
>
> NOTE:
> The userfaultfd write protection feature does not support THP for now,
> Before taking snapshot, please disable THP by:
> echo never > /sys/kernel/mm/transparent_hugepage/enabled
>
> TODO:
> - Reduce the influence for VM while taking snapshot
>
> zhanghailiang (13):
>   postcopy/migration: Split fault related state into struct
>     UserfaultState
>   migration: Allow the migrate command to work on file: urls
>   migration: Allow -incoming to work on file: urls
>   migration: Create a snapshot thread to realize saving memory snapshot
>   migration: implement initialization work for snapshot
>   QEMUSizedBuffer: Introduce two help functions for qsb
>   savevm: Split qemu_savevm_state_complete_precopy() into two helper
>     functions
>   snapshot: Save VM's device state into snapshot file
>   migration/postcopy-ram: fix some helper functions to support
>     userfaultfd write-protect
>   snapshot: Enable the write-protect notification capability for VM's
>     RAM
>   snapshot/migration: Save VM's RAM into snapshot file
>   migration/ram: Fix some helper functions' parameter to use
>     PageSearchStatus
>   snapshot: Remove page's write-protect and copy the content during
>     setup stage
>
>  include/migration/migration.h     |  41 +++++--
>  include/migration/postcopy-ram.h  |   9 +-
>  include/migration/qemu-file.h     |   3 +-
>  include/qemu/typedefs.h           |   1 +
>  include/sysemu/sysemu.h           |   3 +
>  linux-headers/linux/userfaultfd.h |  21 +++-
>  migration/fd.c                    |  51 ++++++++-
>  migration/migration.c             | 101 ++++++++++++++++-
>  migration/postcopy-ram.c          | 229 
> ++++++++++++++++++++++++++++----------
>  migration/qemu-file-buf.c         |  61 ++++++++++
>  migration/ram.c                   | 104 ++++++++++++-----
>  migration/savevm.c                |  90 ++++++++++++---
>  trace-events                      |   1 +
>  13 files changed, 587 insertions(+), 128 deletions(-)
>
> --
> 1.8.3.1
>
>
>

Hi Hailiang,

Can I get the status of this patch series ? I cannot find a v2.
About TCG limitation, is KVM support on a TODO list or is there a
strong technical barrier ?

Thanks,
Baptiste