Re: [Qemu-devel] [PATCH v4] XBZRLE delta for live migration of large memory apps

[Alexander Graf]

On 08.08.2011, at 15:29, Anthony Liguori wrote:

> On 08/08/2011 03:42 AM, Shribman, Aidan wrote:
>> Subject: [PATCH v4] XBZRLE delta for live migration of large memory apps
>> From: Aidan Shribman<address@hidden>
>> 
>> By using XBZRLE (Xor Binary Zero Run-Length-Encoding) we can reduce VM 
>> downtime
>> and total live-migration time of VMs running memory write intensive workloads
>> typical of large enterprise applications such as SAP ERP Systems, and 
>> generally
>> speaking for any application with a sparse memory update pattern.
>> 
>> On the sender side XBZRLE is used as a compact delta encoding of page 
>> updates,
>> retrieving the old page content from an LRU cache (default size of 64 MB). 
>> The
>> receiving side uses the existing page content and XBZRLE to decode the new 
>> page
>> content.
>> 
>> Work was originally based on research results published VEE 2011: Evaluation 
>> of
>> Delta Compression Techniques for Efficient Live Migration of Large Virtual
>> Machines by Benoit, Svard, Tordsson and Elmroth. Additionally the delta 
>> encoder
>> XBRLE was improved further using XBZRLE instead.
>> 
>> XBZRLE has a sustained bandwidth of 2-2.5 GB/s for typical workloads making 
>> it
>> ideal for in-line, real-time encoding such as is needed for live-migration.
>> 
>> A typical usage scenario:
>>     {qemu} migrate_set_cachesize 256m
>>     {qemu} migrate -x -d tcp:destination.host:4444
>>     {qemu} info migrate
>>     ...
>>     transferred ram-duplicate: A kbytes
>>     transferred ram-duplicate: B pages
>>     transferred ram-normal: C kbytes
>>     transferred ram-normal: D pages
>>     transferred ram-xbrle: E kbytes
>>     transferred ram-xbrle: F pages
>>     overflow ram-xbrle: G pages
>>     cache-hit ram-xbrle: H pages
>>     cache-lookup ram-xbrle: J pages
>> 
>> Testing: live migration with XBZRLE completed in 110 seconds, without live
>> migration was not able to complete.
>> 
>> A simple synthetic memory r/w load generator:
>> ..    include<stdlib.h>
>> ..    include<stdio.h>
>> ..    int main()
>> ..    {
>> ..        char *buf = (char *) calloc(4096, 4096);
>> ..        while (1) {
>> ..            int i;
>> ..            for (i = 0; i<  4096 * 4; i++) {
>> ..                buf[i * 4096 / 4]++;
>> ..            }
>> ..            printf(".");
>> ..        }
>> ..    }
>> 
>> Signed-off-by: Benoit Hudzia<address@hidden>
>> Signed-off-by: Petter Svard<address@hidden>
>> Signed-off-by: Aidan Shribman<address@hidden>
> 
> One thing that strikes me about this algorithm is that it's very good for a 
> particular type of workload--shockingly good really.
> 
> I think workload aware migration compression is possible for a lot of 
> different types of workloads.  That makes me a bit wary of QEMU growing quite 
> a lot of compression mechanisms.
> 
> It makes me think that this logic may really belong at a higher level where 
> more information is known about the workload.  For instance, I can imagine 
> XBZRLE living in something like libvirt.
> 
> Today, parsing migration traffic is pretty horrible but I think we're pretty 
> strongly committed to fixing that in 1.0.  That makes me wonder if it would 
> be nicer architecturally for a higher level tool to own something like this.
> 
> Originally, when I added migration, I had the view that we would have 
> transport plugins based on the exec: protocol.  That hasn't really happened 
> since libvirt really owns migration but I think having XBZRLE as a transport 
> plugin for libvirt is something worth considering.
> 
> I'm curious what people think about this type of approach.  CC'ing libvirt to 
> get their input.

In general, I believe it's a good idea to keep looking at libvirt as a vm 
management layer and only a vm management layer. Directly working with the 
migration protocol basically ties us to libvirt if we want to do migration, 
killing competition in the management stack. Just look at how xm is tied to xen 
- it's one of the major points I dislike about it :).


Alex