Re: [Qemu-devel] [RFC] postcopy livemigration proposal

[Dor Laor]

On 08/08/2011 03:32 PM, Anthony Liguori wrote:
> On 08/08/2011 04:20 AM, Dor Laor wrote:
> > On 08/08/2011 06:24 AM, Isaku Yamahata wrote:
> > > This mail is on "Yabusame: Postcopy Live Migration for Qemu/KVM"
> > > on which we'll give a talk at KVM-forum.
> > > The purpose of this mail is to letting developers know it in advance
> > > so that we can get better feedback on its design/implementation approach
> > > early before our starting to implement it.
> > >
> > >
> > > Background
> > > ==========
> > > * What's is postcopy livemigration
> > > It is is yet another live migration mechanism for Qemu/KVM, which
> > > implements the migration technique known as "postcopy" or "lazy"
> > > migration. Just after the "migrate" command is invoked, the execution
> > > host of a VM is instantaneously switched to a destination host.
> > >
> > > The benefit is, total migration time is shorter because it transfer
> > > a page only once. On the other hand precopy may repeat sending same
> > > pages
> > > again and again because they can be dirtied.
> > > The switching time from the source to the destination is several
> > > hunderds mili seconds so that it enables quick load balancing.
> > > For details, please refer to the papers.
> > >
> > > We believe this is useful for others so that we'd like to merge this
> > > feature into the upstream qemu/kvm. The existing implementation that
> > > we have right now is very ad-hoc because it's for academic research.
> > > For the upstream merge, we're starting to re-design/implement it and
> > > we'd like to get feedback early. Although many
> > > improvements/optimizations
> > > are possible, we should implement/merge the simple/clean, but extensible
> > > as well, one at first and then improve/optimize it later.
> > >
> > > postcopy livemigration will be introduced as optional feature. The
> > > existing
> > > precopy livemigration remains as default behavior.
> > >
> > >
> > > * related links:
> > > project page
> > > http://sites.google.com/site/grivonhome/quick-kvm-migration
> > >
> > > Enabling Instantaneous Relocation of Virtual Machines with a
> > > Lightweight VMM Extension,
> > > (proof-of-concept, ad-hoc prototype. not a new design)
> > > http://grivon.googlecode.com/svn/pub/docs/ccgrid2010-hirofuchi-paper.pdf
> > > http://grivon.googlecode.com/svn/pub/docs/ccgrid2010-hirofuchi-talk.pdf
> > >
> > > Reactive consolidation of virtual machines enabled by postcopy live
> > > migration
> > > (advantage for VM consolidation)
> > > http://portal.acm.org/citation.cfm?id=1996125
> > > http://www.emn.fr/x-info/ascola/lib/exe/fetch.php?media=internet:vtdc-postcopy.pdf
> > >
> > >
> > >
> > > Qemu wiki
> > > http://wiki.qemu.org/Features/PostCopyLiveMigration
> > >
> > >
> > > Design/Implementation
> > > =====================
> > > The basic idea of postcopy livemigration is to use a sort of distributed
> > > shared memory between the migration source and destination.
> > >
> > > The migration procedure looks like
> > > - start migration
> > > stop the guest VM on the source and send the machine states except
> > > guest RAM to the destination
> > > - resume the guest VM on the destination without guest RAM contents
> > > - Hook guest access to pages, and pull page contents from the source
> > > This continues until all the pages are pulled to the destination
> > >
> > > The big picture is depicted at
> > > http://wiki.qemu.org/File:Postcopy-livemigration.png
> >
> > That's terrific (nice video also)!
> > Orit and myself had the exact same idea too (now we can't patent it..).
> >
> > Advantages:
> > - No down time due to memory copying.
>
> But non-deterministic down time due to network latency while trying to
> satisfy a page fault.

True but it is possible to limit it with some dedicated network or 
bandwidth reservation.

> > - Efficient, reduce needed traffic no need to re-send pages.
>
> It's not quite that simple. Post-copy needs to introduce a protocol
> capable of requesting pages.

Just another subsection.. (kidding), still it shouldn't be too 
complicated, just an offset+pagesize and return page_content/error

> I think in presenting something like this, it's important to collect
> quite a bit of performance data. I'd suggest doing runs while running
> jitterd in the guest to attempt to quantify the actual downtime
> experienced too.
>
> http://git.codemonkey.ws/cgit/jitterd.git/

and also comparing the speed that it takes for various benchmarks like 
iozone/netperf/linpack/..

> There's a lot of potential in something like this, but it's not obvious
> to me whether it's a net win. Should make for a very interesting
> presentation :-)
>
> > - Reduce overall RAM consumption of the source and destination
> > as opposed from current live migration (both the source and the
> > destination allocate the memory until the live migration
> > completes). We can free copied memory once the destination guest
> > received it and save RAM.
> > - Increase parallelism for SMP guests we can have multiple
> > virtual CPU handle their demand paging . Less time to hold a
> > global lock, less thread contention.
> > - Virtual machines are using more and more memory resources ,
> > for a virtual machine with very large working set doing live
> > migration with reasonable down time is impossible today.
>
> This is really just a limitation of our implementation. In theory,
> pre-copy allows you to exert fine grain resource control over the guest
> which you can use to encourage convergence.

But a very large guest w/ large working set that changes more frequent 
than the network bandwidth might always need huge down time with the 
current system.

> > Disadvantageous:
> > - During the live migration the guest will run slower than in
> > today's live migration. We need to remember that even today
> > guests suffer from performance penalty on the source during the
> > COW stage (memory copy).
> > - Failure of the source or destination or the network will cause
> > us to lose the running virtual machine. Those failures are very
> > rare.
> > In case there is shared storage we can store a copy of the
> > memory there , that can be recovered in case of such failure .
> >
> > Overall, it looks like a better approach for the vast majority of cases.
> > Hope it will get merged to kvm and become the default way.
>
> One thing I think we need to do is put together a live migration
> roadmap. We've got a lot of invasive efforts underway with live
> migration and I fear that without some planning and serialization, some
> of this useful work with get lost.

Some of them are parallel. I think all the readers here agree that post 
copy migration should be an option while we need to maintain the current 
one.

In addition to that there is the switch to separate thread which is 
standalone, and the vmstate vs QOM/visitor work.

Lastly the compression work..

> Regards,
>
> Anthony Liguori
>
> > > There are several design points.
> > > - who takes care of pulling page contents.
> > > an independent daemon vs a thread in qemu
> > > The daemon approach is preferable because an independent daemon would
> > > easy for debug postcopy memory mechanism without qemu.
> > > If required, it wouldn't be difficult to convert a daemon into
> > > a thread in qemu
> > >
> > > - connection between the source and the destination
> > > The connection for live migration can be re-used after sending machine
> > > state.
> > >
> > > - transfer protocol
> > > The existing protocol that exists today can be extended.
> > >
> > > - hooking guest RAM access
> > > Introduce a character device to handle page fault.
> > > When page fault occurs, it queues page request up to user space daemon
> > > at the destination. And the daemon pulls page contents from the source
> > > and serves it into the character device. Then the page fault is
> > > resovlved.
> >
> > Isn't there a simpler way of using madvise verb to mark that the
> > destination guest RAM will need paging?
> >
> > Cheers and looking forward to the presentation over the kvm forum,
> > Dor
> >
> > > * More on hooking guest RAM access
> > > There are several candidate for the implementation. Our preference is
> > > character device approach.
> > >
> > > - inserting hooks into everywhere in qemu/kvm
> > > This is impractical
> > >
> > > - backing store for guest ram
> > > a block device or a file can be used to back guest RAM.
> > > Thus hook the guest ram access.
> > >
> > > pros
> > > - new device driver isn't needed.
> > > cons
> > > - future improvement would be difficult
> > > - some KVM host feature(KSM, THP) wouldn't work
> > >
> > > - character device
> > > qemu mmap() the dedicated character device, and then hook page fault.
> > >
> > > pros
> > > - straght forward approach
> > > - future improvement would be easy
> > > cons
> > > - new driver is needed
> > > - some KVM host feature(KSM, THP) wouldn't work
> > > They checks if a given VMA is anonymous. This can be fixed.
> > >
> > > - swap device
> > > When creating guest, it is set up as if all the guest RAM is swapped out
> > > to a dedicated swap device, which may be nbd disk (or some kind of user
> > > space block device, BUSE?).
> > > When the VM tries to access memory, swap-in is triggered and IO to the
> > > swap device is issued. Then the IO to swap is routed to the daemon
> > > in user space with nbd protocol (or BUSE, AOE, iSCSI...). The daemon
> > > pulls
> > > pages from the migration source and services the IO request.
> > >
> > > pros
> > > - After the page transfer is complete, everything is same as normal
> > > case.
> > > - no new device driver isn't needed
> > > cons
> > > - future improvement would be difficult
> > > - administration: setting up nbd, swap device
> > >
> > > Thanks in advance