[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [Qemu-devel] [PATCH v6 00/47] Postcopy implementation
From: |
Li, Liang Z |
Subject: |
Re: [Qemu-devel] [PATCH v6 00/47] Postcopy implementation |
Date: |
Thu, 30 Apr 2015 01:09:35 +0000 |
> * Li, Liang Z (address@hidden) wrote:
> > Hi David,
> >
> > I have tired your v6 postcopy patches and found it doesn't work. When
> > I tried to start the postcopy in live migration, some errors were printed. I
> just did the following things:
> >
> > On destination side, started the qemu like this:
> >
> > /root/vt-sync/post_copy_v6_qemu.git/x86_64-softmmu/qemu-system-
> x86_64
> > -enable-kvm -smp 2 -m 1024 -net none /mnt/jinshi_ia32e_rhel6u5.qcow2
> > -monitor stdio -incoming tcp:0:4444
> >
> > On source side, started the qemu like this:
> >
> > /root/vt-sync/post_copy_v6_qemu.git/x86_64-softmmu/qemu-system-
> x86_64
> > -enable-kvm -smp 2 -m 1024 -net none /mnt/jinshi_ia32e_rhel6u5.qcow2
> > -monitor stdio
> >
> > and then
> > (qemu) migrate_set_capability x-postcopy-ram on
> >
> > When I started the post copy with
> > (qemu) migrate -d tcp:localhost:4444
> >
> > I got the error message on the source side:
> >
> > (qemu) qemu-system-x86_64: socket_writev_buffer: Got err=104 for
> (131552/-1)
> > qemu-system-x86_64: RP: Received invalid message 0x0000 length
> 0x0000
> >
> > and the following error on the destination side:
> >
> > (qemu) qemu-system-x86_64: postcopy_ram_supported_by_host: No OS
> > support
> > qemu-system-x86_64: load of migration failed: Operation not permitted
>
> OK, the important error here is:
> postcopy_ram_supported_by_host: No OS support
>
> that's saying that the destination OS either:
> 1) The kernel isn't the correct kernel with Andrea's userfault code
> compiled
> in
> (check that userfaultfd is configured into the kernel as well)
> 2) That when you built the QEMU it didn't find the syscall definition for
> the
> userfaultfd in the header as it compiled it.
>
> I think from that error it is (2) - so make sure that when you built the qemu
> that you're using the headers from that kernel, or use the extra-cflags hack
> that I mentioned in the cover letter.
>
> Note that you need to use the kernel tree which I point to in the first
> message.
> (The older kernel from v5 wont work).
>
Thanks Dave, I will retry according to your suggestion.
> Dave
> P.S. I'm on holiday this week, so not checking work email much.
>
> >
> >
> > the dmesg printed:
> > [ 233.456545] kvm: zapping shadow pages for mmio generation
> > wraparound [ 239.785916] kvm [11926]: vcpu0 disabled perfctr wrmsr:
> > 0xc1 data 0xabcd
> >
> >
> > The v5 patches have no such errors. Do you have any suggestion?
> >
> > Liang
> >
> >
> > > -----Original Message-----
> > > From: address@hidden
> > > [mailto:qemu-
> > > address@hidden On Behalf Of Dr.
> > > devel-bounces+David Alan
> > > Gilbert (git)
> > > Sent: Wednesday, April 15, 2015 1:03 AM
> > > To: address@hidden
> > > Cc: address@hidden; address@hidden;
> > > address@hidden; address@hidden; address@hidden;
> > > address@hidden; address@hidden
> > > Subject: [Qemu-devel] [PATCH v6 00/47] Postcopy implementation
> > >
> > > From: "Dr. David Alan Gilbert" <address@hidden>
> > >
> > > This is the 6th cut of my version of postcopy; it is designed for
> > > use with the Linux kernel additions posted by Andrea Arcangeli here:
> > >
> > > git clone --reference linux -b userfault18
> > > git://git.kernel.org/pub/scm/linux/kernel/git/andrea/aa.git
> > >
> > > (Note this is a different API from the last version)
> > >
> > > This qemu series can be found at:
> > >
> > > https://github.com/orbitfp7/qemu.git
> > > on the wp3-postcopy-v6 tag.
> > >
> > > It addresses some but not yet all of the previous review comments;
> > > however there are a couple of large simplifications, so it seems
> > > worth posting to meet the new kernel API and to stop people reviewing
> deadcode.
> > >
> > > Note: That the userfaultfd.h header is no longer included in this
> > > tree:
> > > - if you're building with the appropriate kernel headers it should
> > > find it
> > > - if you're building on a host that doesn't have the kernel headers
> > > installed in the right place then:
> > > configure with: --extra-cflags="-D__NR_userfaultfd=323"
> > > cp include/uapi/linux/userfaultfd.h into somewhere in the
> > > include
> > > path, e.g. /usr/local/include/linux
> > >
> > > v6
> > > Removed the PMI bitmaps
> > > - Andrea updated the kernel API so that userspace doesn't
> > > need to do wakeups, and thus QEMU doesn't need to keep
> > > track of which pages it's received; there is a price - which
> > > is we end up sending more dupes to the source, but it simplifies
> > > stuff a lot and makes the normal paths a lot quicker.
> > > (10s of line change in kernel, 10%-ish simplification in this
> > > code!)
> > > Changed discard message format to a simpler start/end address scheme
> > > and rework discard and chunking code to work in long's to match
> bitmap
> > > 'qemu_get_buffer_less_copy' for postcopy pages
> > > - avoids a userspace copy since the kernel now does it
> > > - the new qemufile interface might also be useful for other places
> > > that
> > > don't need a copy (maybe xbzrle?)
> > > Changed the blockingness of the incoming fd
> > > it was incorrectly blocking during the precopy phase after a
> > > postcopy
> was
> > > enabled, causing the HMP to be unavailable. It's now blocking only
> once
> > > the postcopy thread starts up, since it's not a coroutine it can't
> > > deal
> > > with the yields in qemu_file.
> > > An error on the return-path now marks the migration as failed
> > >
> > > Fixups from Dave Gibson's comments
> > > Removed can_postcopy, renamed save_complete to
> > > save_complete_precopy
> > > added save_complete_postcopy
> > > Simplified loadvm loop exits
> > > discard message format changes above
> > > and many more smaller changes.
> > >
> > > small fixups for RCU
> > >
> > >
> > > This work has been partially funded by the EU Orbit project:
> > > see http://www.orbitproject.eu/about/
> > >
> > > TODO:
> > > The major work is to rework the page send/receive loops so that
> supporting
> > > larger host pages doesn't make it quite as messy.
> > >
> > > Dr. David Alan Gilbert (47):
> > > Start documenting how postcopy works.
> > > Split header writing out of qemu_savevm_state_begin
> > > qemu_ram_foreach_block: pass up error value, and down the ramblock
> > > name
> > > Add qemu_get_counted_string to read a string prefixed by a count byte
> > > Create MigrationIncomingState
> > > Provide runtime Target page information
> > > Move copy out of qemu_peek_buffer
> > > Add qemu_get_buffer_less_copy to avoid copies some of the time
> > > Add wrapper for setting blocking status on a QEMUFile
> > > Rename save_live_complete to save_live_complete_precopy
> > > Return path: Open a return path on QEMUFile for sockets
> > > Return path: socket_writev_buffer: Block even on non-blocking fd's
> > > Migration commands
> > > Return path: Control commands
> > > Return path: Send responses from destination to source
> > > Return path: Source handling of return path
> > > ram_debug_dump_bitmap: Dump a migration bitmap as text
> > > Move loadvm_handlers into MigrationIncomingState
> > > Rework loadvm path for subloops
> > > Add migration-capability boolean for postcopy-ram.
> > > Add wrappers and handlers for sending/receiving the postcopy-ram
> > > migration messages.
> > > MIG_CMD_PACKAGED: Send a packaged chunk of migration stream
> > > migrate_init: Call from savevm
> > > Modify save_live_pending for postcopy
> > > postcopy: OS support test
> > > migrate_start_postcopy: Command to trigger transition to postcopy
> > > MIGRATION_STATUS_POSTCOPY_ACTIVE: Add new migration state
> > > Add qemu_savevm_state_complete_postcopy
> > > Postcopy: Maintain sentmap and calculate discard
> > > postcopy: Incoming initialisation
> > > postcopy: ram_enable_notify to switch on userfault
> > > Postcopy: Postcopy startup in migration thread
> > > Postcopy end in migration_thread
> > > Page request: Add MIG_RP_MSG_REQ_PAGES reverse command
> > > Page request: Process incoming page request
> > > Page request: Consume pages off the post-copy queue
> > > postcopy_ram.c: place_page and helpers
> > > Postcopy: Use helpers to map pages during migration
> > > qemu_ram_block_from_host
> > > Don't sync dirty bitmaps in postcopy
> > > Host page!=target page: Cleanup bitmaps
> > > Postcopy; Handle userfault requests
> > > Start up a postcopy/listener thread ready for incoming page data
> > > postcopy: Wire up loadvm_postcopy_handle_ commands
> > > End of migration for postcopy
> > > Disable mlock around incoming postcopy
> > > Inhibit ballooning during postcopy
> > >
> > > arch_init.c | 868
> ++++++++++++++++++++++++++++++++++++---
> > > balloon.c | 11 +
> > > docs/migration.txt | 167 ++++++++
> > > exec.c | 74 +++-
> > > hmp-commands.hx | 15 +
> > > hmp.c | 7 +
> > > hmp.h | 1 +
> > > hw/ppc/spapr.c | 2 +-
> > > hw/virtio/virtio-balloon.c | 4 +-
> > > include/exec/cpu-all.h | 2 -
> > > include/exec/cpu-common.h | 7 +-
> > > include/migration/migration.h | 126 +++++-
> > > include/migration/postcopy-ram.h | 88 ++++
> > > include/migration/qemu-file.h | 15 +-
> > > include/migration/vmstate.h | 10 +-
> > > include/qemu/typedefs.h | 5 +
> > > include/sysemu/balloon.h | 2 +
> > > include/sysemu/sysemu.h | 45 +-
> > > migration/Makefile.objs | 2 +-
> > > migration/block.c | 9 +-
> > > migration/migration.c | 743
> +++++++++++++++++++++++++++++++--
> > > migration/postcopy-ram.c | 715
> > > ++++++++++++++++++++++++++++++++
> > > migration/qemu-file-unix.c | 106 ++++-
> > > migration/qemu-file.c | 100 ++++-
> > > migration/rdma.c | 4 +-
> > > migration/vmstate.c | 5 +-
> > > qapi-schema.json | 19 +-
> > > qmp-commands.hx | 19 +
> > > savevm.c | 809
> > > ++++++++++++++++++++++++++++++++----
> > > trace-events | 77 +++-
> > > 30 files changed, 3832 insertions(+), 225 deletions(-) create mode
> > > 100644 include/migration/postcopy-ram.h create mode 100644
> > > migration/postcopy- ram.c
> > >
> > > --
> > > 2.1.0
> > >
> >
> --
> Dr. David Alan Gilbert / address@hidden / Manchester, UK
- [Qemu-devel] [PATCH v6 41/47] Host page!=target page: Cleanup bitmaps, (continued)
- [Qemu-devel] [PATCH v6 41/47] Host page!=target page: Cleanup bitmaps, Dr. David Alan Gilbert (git), 2015/04/14
- [Qemu-devel] [PATCH v6 43/47] Start up a postcopy/listener thread ready for incoming page data, Dr. David Alan Gilbert (git), 2015/04/14
- [Qemu-devel] [PATCH v6 31/47] postcopy: ram_enable_notify to switch on userfault, Dr. David Alan Gilbert (git), 2015/04/14
- [Qemu-devel] [PATCH v6 42/47] Postcopy; Handle userfault requests, Dr. David Alan Gilbert (git), 2015/04/14
- [Qemu-devel] [PATCH v6 46/47] Disable mlock around incoming postcopy, Dr. David Alan Gilbert (git), 2015/04/14
- [Qemu-devel] [PATCH v6 45/47] End of migration for postcopy, Dr. David Alan Gilbert (git), 2015/04/14
- [Qemu-devel] [PATCH v6 44/47] postcopy: Wire up loadvm_postcopy_handle_ commands, Dr. David Alan Gilbert (git), 2015/04/14
- [Qemu-devel] [PATCH v6 47/47] Inhibit ballooning during postcopy, Dr. David Alan Gilbert (git), 2015/04/14
- Re: [Qemu-devel] [PATCH v6 00/47] Postcopy implementation, Li, Liang Z, 2015/04/27