Re: [Qemu-devel] [PATCH v5 0/9] XBZRLE delta for live migration of large memory apps

[Orit Wasserman]

On 01/03/2012 06:32 PM, Anthony Liguori wrote:
> On 01/03/2012 09:34 AM, Orit Wasserman wrote:
>> Changes from v4:
>> 1) Rebase
>> 2) divide patch into 9 patches
>> 3) move memory allocation into cache_insert
>>
>> By using XBZRLE (Xor Binary Zero Run-Length-Encoding) we can reduce VM 
>> downtime
>> and total live-migration time of VMs running memory write intensive workloads
>> typical of large enterprise applications such as SAP ERP Systems, and 
>> generally
>> speaking for any application with a sparse memory update pattern.
>>
>> On the sender side XBZRLE is used as a compact delta encoding of page 
>> updates,
>> retrieving the old page content from an LRU cache (default size of 64 MB). 
>> The
>> receiving side uses the existing page content and XBZRLE to decode the new 
>> page
>> content.
>>
>> Work was originally based on research results published VEE 2011: Evaluation 
>> of
>> Delta Compression Techniques for Efficient Live Migration of Large Virtual
>> Machines by Benoit, Svard, Tordsson and Elmroth. Additionally the delta 
>> encoder
>> XBRLE was improved further using XBZRLE instead.
>>
>> XBZRLE has a sustained bandwidth of 2-2.5 GB/s for typical workloads making 
>> it
>> ideal for in-line, real-time encoding such as is needed for live-migration.
>>
>> A typical usage scenario:
>>      {qemu} migrate_set_cachesize 256m
> 
> How does one intelligently choose a cache size?  How bad is it to choose the 
> wrong cache size?

If the cache you chose is was too big than you just wasted some memory 
(probably small amount for extra 100M it is around 12K).
If the cache is too small than you are not using the full capability , if you 
will enlarge the cache
you can compress more pages. you still get performance benefits but you will 
get more if had chosen bigger cache. 
> 
>>      {qemu} migrate -x -d tcp:destination.host:4444
> 
> We need to make this a negotiated feature, not a requirement for the user to 
> figure out.
> 
> Please go back to the previous thread for more details on this.

I will update the patch for the feature to be negotiable. 
> 
> Regards,
> 
> Anthony Liguori
> 
>>      {qemu} info migrate
>>      ...
>>      transferred ram-duplicate: A kbytes
>>      transferred ram-duplicate: B pages
>>      transferred ram-normal: C kbytes
>>      transferred ram-normal: D pages
>>      transferred ram-xbrle: E kbytes
>>      transferred ram-xbrle: F pages
>>      overflow ram-xbrle: G pages
>>      cache-hit ram-xbrle: H pages
>>      cache-lookup ram-xbrle: J pages
>>
>> Testing: live migration with XBZRLE completed in 110 seconds, without live
>> migration was not able to complete.
>>
>> A simple synthetic memory r/w load generator:
>> ..    include<stdlib.h>
>> ..    include<stdio.h>
>> ..    int main()
>> ..    {
>> ..        char *buf = (char *) calloc(4096, 4096);
>> ..        while (1) {
>> ..            int i;
>> ..            for (i = 0; i<  4096 * 4; i++) {
>> ..                buf[i * 4096 / 4]++;
>> ..            }
>> ..            printf(".");
>> ..        }
>> ..    }
>>
>> Signed-off-by: Benoit Hudzia<address@hidden>
>> Signed-off-by: Petter Svard<address@hidden>
>> Signed-off-by: Aidan Shribman<address@hidden>
>>
>>   arch_init.c       |  546 
>> +++++++++++++++++++++++++++++++++++++++++++++++++----
>>   block-migration.c |    4 +-
>>   hmp-commands.hx   |   34 +++-
>>   hw/hw.h           |    4 +-
>>   migration.c       |   51 +++++-
>>   migration.h       |   19 ++
>>   qmp-commands.hx   |   44 ++++-
>>   savevm.c          |   11 +-
>>   sysemu.h          |    4 +-
>>   9 files changed, 653 insertions(+), 64 deletions(-)
>>
>