Changes from v8:
Implement more effiecent cache_resize method
fix set_cachesize command comments
Changes from v7:
Copy current page before encoding it, this will prevents page content
change during the encoding.
Allow changing the cache size during an active migration.
Fix comments by Avi.
Changes from v6:
1) add assert checks to ULEB encoding/decoding
2) no need to send last zero run
Changes from v5:
1) Add migration capabilities
2) Use ULEB to encode run length
3) Do not send unmodified (dirty) page
3) Fix other patch comments
Using GCache or GHashTable requires allocating new buffer on every content
change,
so I decided to keep the simple cache implementation.
Changes from v4:
1) Rebase
2) divide patch into 9 patches
3) move memory allocation into cache_insert
Future work :
Use SSE for encoding.
Page ranking acording to their dirty rate and automatic
activation/deactivation of the feature - will be sent in a separate patch
series.
By using XBZRLE (Xor Based Zero Run Length Encoding) we can reduce VM downtime
and total live-migration time of VMs running memory write intensive workloads
typical of large enterprise applications such as SAP ERP Systems, and generally
speaking for any application with a sparse memory update pattern.
The compression format uses the fact that we will have many zero (zero
represents
an unchanged value).
We repesent the page data delta by zero and non zero runs.
We represent a zero run with it's length (in bytes).
We represent a non zero run with it's length (in bytes) and the data.
The run length is encoded using ULEB128 (http://en.wikipedia.org/wiki/LEB128)
page = zrun nzrun
| zrun nzrun page
zrun = length
nzrun = length byte...
length = uleb128 encoded integer
On the sender side XBZRLE is used as a compact delta encoding of page updates,
retrieving the old page content from an LRU cache (default size of 512 MB). The
receiving side uses the existing page content and XBZRLE to decode the new page
content.
This is a more compact way to store the delta than the previous version.
This work was originally based on research results published VEE 2011:
Evaluation of
Delta Compression Techniques for Efficient Live Migration of Large Virtual
Machines by Benoit, Svard, Tordsson and Elmroth. Additionally the delta encoder
XBRLE was improved further using XBZRLE instead.
XBZRLE has a sustained bandwidth of 2-2.5 GB/s for typical workloads making it
ideal for in-line, real-time encoding such as is needed for live-migration.
A typical usage scenario:
{qemu} migrate_set_cachesize 256m
{qemu} migrate -d -x tcp:destination.host:4444
{qemu} info migrate
...
transferred ram-duplicate: A kbytes
transferred ram-normal: B kbytes
transferred ram-xbrle: C kbytes
overflow ram-xbrle: D pages
cache-miss ram-xbrle: E pages
Testing: live migration with XBZRLE completed in 110 seconds, without live
migration was not able to complete.
A simple synthetic memory r/w load generator:
.. include<stdlib.h>
.. include<stdio.h>
.. int main()
.. {
.. char *buf = (char *) calloc(4096, 4096);
.. while (1) {
.. int i;
.. for (i = 0; i< 4096 * 4; i++) {
.. buf[i * 4096 / 4]++;
.. }
.. printf(".");
.. }
.. }
Signed-off-by: Benoit Hudzia<address@hidden>
Signed-off-by: Petter Svard<address@hidden>
Signed-off-by: Aidan Shribman<address@hidden>
Orit Wasserman (10):
Add cache handling functions
Add uleb encoding/decoding functions
Add save_block_hdr function
Add host_from_stream_offset_versioned function
Add MigrationParams structure
Add XBZRLE to ram_save_block and ram_save_live
Add XBZRLE option to migrate command
Add migration capabilites
Add set_cachesize command
Add XBZRLE statistics
arch_init.c | 521 ++++++++++++++++++++++++++++++++++++++++++++++++++---
block-migration.c | 8 +-
hmp-commands.hx | 35 +++-
hmp.c | 48 +++++-
hmp.h | 2 +
migration.c | 92 +++++++++-
migration.h | 32 +++-
monitor.c | 7 +
qapi-schema.json | 64 +++++++-
qemu-common.h | 1 +
qmp-commands.hx | 78 ++++++++-
savevm.c | 133 +++++++++++++-
sysemu.h | 3 +-
vmstate.h | 2 +-
14 files changed, 969 insertions(+), 57 deletions(-)