Re: [Qemu-devel] [PATCH v3 0/4] migration: skip scanning and migrating ram pages released by virtio-balloon driver.

[Jitendra Kolhe]

ping ...
also had received some bounce back from few individual email-ids, 
so consider this one as resend.

Thanks,
- Jitendra

On 5/30/2016 4:19 PM, Jitendra Kolhe wrote:
> ping...
> for entire v3 version of the patchset.
> http://patchwork.ozlabs.org/project/qemu-devel/list/?submitter=68462
> 
> - Jitendra
> 
> On Wed, May 18, 2016 at 4:50 PM, Jitendra Kolhe <address@hidden> wrote:
>> While measuring live migration performance for qemu/kvm guest, it was 
>> observed
>> that the qemu doesn’t maintain any intelligence for the guest ram pages 
>> released
>> by the guest balloon driver and treat such pages as any other
>> normal guest ram pages. This has direct impact on overall migration time for
>> the guest which has released (ballooned out) memory to the host.
>>
>> In case of large systems, where we can configure large guests with 1TB and
>> with considerable amount of memory released by balloon driver to the host,
>> the migration time gets worse.
>>
>> The solution proposed below is local to qemu (and does not require any
>> modification to Linux kernel or any guest driver). We have verified the fix
>> for large guests =1TB on HPE Superdome X (which can support up to 240 cores
>> and 12TB of memory).
>>
>> During live migration, as part of first iteration in ram_save_iterate() ->
>> ram_find_and_save_block () will try to migrate ram pages even if they are
>> released by vitrio-balloon driver (balloon inflate). Although these pages
>> which are returned to the host by virtio-balloon driver are zero pages,
>> the migration algorithm will still end up scanning the entire page
>> ram_find_and_save_block() -> ram_save_page()/ram_save_compressed_page() ->
>> save_zero_page() -> is_zero_range(). We also end-up sending header 
>> information
>> over network for these pages during migration. This adds to the total
>> migration time.
>>
>> The solution creates a balloon bitmap ramblock as a part of virtio-balloon
>> device initialization. The bits in the balloon bitmap represent a guest ram
>> page of size 1UL << VIRTIO_BALLOON_PFN_SHIFT or 4K. If TARGET_PAGE_BITS <=
>> VIRTIO_BALLOON_PFN_SHIFT, ram_addr offset for the dirty page which is used by
>> dirty page bitmap during migration is checked against the balloon bitmap as
>> is, if the bit is set in the balloon bitmap, the corresponding ram page will 
>> be
>> excluded from scanning and sending header information during migration. In 
>> case
>> TARGET_PAGE_BITS > VIRTIO_BALLOON_PFN_SHIFT for a given dirty page ram_addr,
>> all sub-pages of 1UL << VIRTIO_BALLOON_PFN_SHIFT size should be ballooned out
>> to avoid zero page scan and sending header information.
>>
>> The bitmap represents entire guest ram memory till max configured memory.
>> Guest ram pages claimed by the virtio-balloon driver will be represented by 1
>> in the bitmap. Since the bitmap is maintained as a ramblock, it’s migrated to
>> target as part migration’s ram iterative and ram complete phase. So that
>> substituent migrations from the target can continue to use optimization.
>>
>> A new migration capability called skip-balloon is introduced. The user can
>> disable the capability in cases where user does not expect much benefit or in
>> case the migration is from an older version.
>>
>> During live migration setup the optimization can be set to disabled state if
>> . no virtio-balloon device is initialized.
>> . skip-balloon migration capability is disabled.
>> . If the guest virtio-balloon driver has not set 
>> VIRTIO_BALLOON_F_MUST_TELL_HOST
>>   flag. Which means the guest may start using a ram pages freed by guest 
>> balloon
>>   driver, even before the host/qemu is aware of it. In such case, the
>>   optimization is disabled so that the ram pages that are being used by the
>>   guest will continue to be scanned and migrated.
>>
>> Balloon bitmap ramblock size is set to zero if the optimization is disabled,
>> to avoid overhead of migrating the bitmap. If the bitmap is not migrated to
>> the target, the destination starts with a fresh bitmap and tracks the
>> ballooning operation thereafter.
>>
>> Jitendra Kolhe (4):
>>   balloon: maintain bitmap for pages released by guest balloon driver.
>>   balloon: add balloon bitmap migration capability and setup bitmap
>>     migration      status.
>>   balloon: reset balloon bitmap ramblock size on source and target.
>>   migration: skip scanning and migrating ram pages released by
>>     virtio-balloon     driver.
>>
>> Changed in v2:
>>  - Resolved compilation issue for qemu-user binaries in exec.c
>>  - Localize balloon bitmap test to save_zero_page().
>>  - Updated version string for newly added migration capability to 2.7.
>>  - Made minor modifications to patch commit text.
>>
>> Changed in v3:
>>  - Add balloon bitmap to RAMBlock.
>>  - Resolve bitmap offset calculation by translating host addr back to a
>>    RAMBlock and ram_addr
>>  - Add balloon bitmap support for case if TARGET_PAGE_BITS
>>    > VIRTIO_BALLOON_PFN_SHIFT.
>>  - Remove dependency of skip-balloon migration capability on postcopy
>>    migration.
>>  - Disable optimization if the guest balloon driver does not support
>>    VIRTIO_BALLOON_F_MUST_TELL_HOST feature.
>>  - Split single patch into 4 small patches.
>>
>>  balloon.c                          | 196 
>> ++++++++++++++++++++++++++++++++++++-
>>  exec.c                             |   6 ++
>>  hw/virtio/virtio-balloon.c         |  42 +++++++-
>>  include/hw/virtio/virtio-balloon.h |   1 +
>>  include/migration/migration.h      |   1 +
>>  include/sysemu/balloon.h           |  13 ++-
>>  migration/migration.c              |   9 ++
>>  migration/ram.c                    |   9 +-
>>  qapi-schema.json                   |   5 +-
>>  9 files changed, 276 insertions(+), 6 deletions(-)
>>
>> --
>> 1.8.3.1
>>
>>