Re: [Qemu-devel] [PATCH v9 00/10] XBZRLE delta for live migration of large memory app

[Orit Wasserman]

On 04/18/2012 08:12 PM, Anthony Liguori wrote:
> Hi,
> 
> Just to avoid any 1.1 migration "surprises", I view this series as 1.2 
> material.  I'm reluctant to make any protocol changes this late in the 
> release process (assuming this series got all the necessary review/ack in the 
> next 1.5 weeks).
> 
> If anyone has a strong feeling differently, now's the time to speak up (as 
> opposed to during the -rc phase).
Hi Anthony,

I was indeed aiming this series for 1.2.

Cheers,
Orit
> 
> Regards,
> 
> Anthony Liguori
> 
> 
> On 04/11/2012 01:49 PM, Orit Wasserman wrote:
>> Changes from v8:
>>     Implement more effiecent cache_resize method
>>     fix set_cachesize command comments
>>
>> Changes from v7:
>>     Copy current page before encoding it, this will prevents page content
>>     change during the encoding.
>>     Allow changing the cache size during an active migration.
>>     Fix comments by Avi.
>>
>> Changes from v6:
>>   1) add assert checks to ULEB encoding/decoding
>>   2) no need to send last zero run
>>     
>> Changes from v5:
>> 1) Add migration capabilities
>> 2) Use ULEB to encode run length
>> 3) Do not send unmodified (dirty) page
>> 3) Fix other patch comments
>>
>> Using GCache or GHashTable requires allocating new buffer on every content 
>> change,
>> so I decided to keep the simple cache implementation.
>>
>> Changes from v4:
>> 1) Rebase
>> 2) divide patch into 9 patches
>> 3) move memory allocation into cache_insert
>>
>> Future work :
>>       Use SSE for encoding.
>>       Page ranking acording to their dirty rate and automatic 
>> activation/deactivation of the feature - will be sent in a separate patch 
>> series.   
>>
>> By using XBZRLE (Xor Based Zero Run Length Encoding) we can reduce VM 
>> downtime
>> and total live-migration time of VMs running memory write intensive workloads
>> typical of large enterprise applications such as SAP ERP Systems, and 
>> generally
>> speaking for any application with a sparse memory update pattern.
>>
>> The compression format uses the fact that we will have many zero (zero 
>> represents
>> an unchanged value).
>> We repesent the page data delta by zero and non zero runs.
>> We represent a zero run with it's length (in bytes).
>> We represent a non zero run with it's length (in bytes) and the data.
>> The run length is encoded using ULEB128 (http://en.wikipedia.org/wiki/LEB128)
>>
>> page = zrun nzrun
>>         | zrun nzrun page
>>
>> zrun = length
>>
>> nzrun = length byte...
>>
>> length = uleb128 encoded integer
>>
>> On the sender side XBZRLE is used as a compact delta encoding of page 
>> updates,
>> retrieving the old page content from an LRU cache (default size of 512 MB). 
>> The
>> receiving side uses the existing page content and XBZRLE to decode the new 
>> page
>> content.
>>
>> This is a more compact way to store the delta than the previous version.
>>
>> This work was originally based on research results published VEE 2011: 
>> Evaluation of
>> Delta Compression Techniques for Efficient Live Migration of Large Virtual
>> Machines by Benoit, Svard, Tordsson and Elmroth. Additionally the delta 
>> encoder
>> XBRLE was improved further using XBZRLE instead.
>>
>> XBZRLE has a sustained bandwidth of 2-2.5 GB/s for typical workloads making 
>> it
>> ideal for in-line, real-time encoding such as is needed for live-migration.
>>
>> A typical usage scenario:
>>      {qemu} migrate_set_cachesize 256m
>>      {qemu} migrate -d -x tcp:destination.host:4444
>>      {qemu} info migrate
>>      ...
>>      transferred ram-duplicate: A kbytes
>>      transferred ram-normal: B kbytes
>>      transferred ram-xbrle: C kbytes
>>      overflow ram-xbrle: D pages
>>      cache-miss ram-xbrle: E pages
>>
>> Testing: live migration with XBZRLE completed in 110 seconds, without live
>> migration was not able to complete.
>>
>> A simple synthetic memory r/w load generator:
>> ..    include<stdlib.h>
>> ..    include<stdio.h>
>> ..    int main()
>> ..    {
>> ..        char *buf = (char *) calloc(4096, 4096);
>> ..        while (1) {
>> ..            int i;
>> ..            for (i = 0; i<  4096 * 4; i++) {
>> ..                buf[i * 4096 / 4]++;
>> ..            }
>> ..            printf(".");
>> ..        }
>> ..    }
>>
>> Signed-off-by: Benoit Hudzia<address@hidden>
>> Signed-off-by: Petter Svard<address@hidden>
>> Signed-off-by: Aidan Shribman<address@hidden>
>>
>> Orit Wasserman (10):
>>    Add cache handling functions
>>    Add uleb encoding/decoding functions
>>    Add save_block_hdr function
>>    Add host_from_stream_offset_versioned function
>>    Add MigrationParams structure
>>    Add XBZRLE to ram_save_block and ram_save_live
>>    Add XBZRLE option to migrate command
>>    Add migration capabilites
>>    Add set_cachesize command
>>    Add XBZRLE statistics
>>
>>   arch_init.c       |  521 
>> ++++++++++++++++++++++++++++++++++++++++++++++++++---
>>   block-migration.c |    8 +-
>>   hmp-commands.hx   |   35 +++-
>>   hmp.c             |   48 +++++-
>>   hmp.h             |    2 +
>>   migration.c       |   92 +++++++++-
>>   migration.h       |   32 +++-
>>   monitor.c         |    7 +
>>   qapi-schema.json  |   64 +++++++-
>>   qemu-common.h     |    1 +
>>   qmp-commands.hx   |   78 ++++++++-
>>   savevm.c          |  133 +++++++++++++-
>>   sysemu.h          |    3 +-
>>   vmstate.h         |    2 +-
>>   14 files changed, 969 insertions(+), 57 deletions(-)
>>
> 
>