Re: [Qemu-devel] [PATCH v3 0/4] migration: skip scanning and migrating ram pages released by virtio-balloon driver.

[Jitendra Kolhe]

ping...
for entire v3 version of the patchset.
http://patchwork.ozlabs.org/project/qemu-devel/list/?submitter=68462

- Jitendra

On Wed, May 18, 2016 at 4:50 PM, Jitendra Kolhe <address@hidden> wrote:
> While measuring live migration performance for qemu/kvm guest, it was observed
> that the qemu doesn’t maintain any intelligence for the guest ram pages 
> released
> by the guest balloon driver and treat such pages as any other
> normal guest ram pages. This has direct impact on overall migration time for
> the guest which has released (ballooned out) memory to the host.
>
> In case of large systems, where we can configure large guests with 1TB and
> with considerable amount of memory released by balloon driver to the host,
> the migration time gets worse.
>
> The solution proposed below is local to qemu (and does not require any
> modification to Linux kernel or any guest driver). We have verified the fix
> for large guests =1TB on HPE Superdome X (which can support up to 240 cores
> and 12TB of memory).
>
> During live migration, as part of first iteration in ram_save_iterate() ->
> ram_find_and_save_block () will try to migrate ram pages even if they are
> released by vitrio-balloon driver (balloon inflate). Although these pages
> which are returned to the host by virtio-balloon driver are zero pages,
> the migration algorithm will still end up scanning the entire page
> ram_find_and_save_block() -> ram_save_page()/ram_save_compressed_page() ->
> save_zero_page() -> is_zero_range(). We also end-up sending header information
> over network for these pages during migration. This adds to the total
> migration time.
>
> The solution creates a balloon bitmap ramblock as a part of virtio-balloon
> device initialization. The bits in the balloon bitmap represent a guest ram
> page of size 1UL << VIRTIO_BALLOON_PFN_SHIFT or 4K. If TARGET_PAGE_BITS <=
> VIRTIO_BALLOON_PFN_SHIFT, ram_addr offset for the dirty page which is used by
> dirty page bitmap during migration is checked against the balloon bitmap as
> is, if the bit is set in the balloon bitmap, the corresponding ram page will 
> be
> excluded from scanning and sending header information during migration. In 
> case
> TARGET_PAGE_BITS > VIRTIO_BALLOON_PFN_SHIFT for a given dirty page ram_addr,
> all sub-pages of 1UL << VIRTIO_BALLOON_PFN_SHIFT size should be ballooned out
> to avoid zero page scan and sending header information.
>
> The bitmap represents entire guest ram memory till max configured memory.
> Guest ram pages claimed by the virtio-balloon driver will be represented by 1
> in the bitmap. Since the bitmap is maintained as a ramblock, it’s migrated to
> target as part migration’s ram iterative and ram complete phase. So that
> substituent migrations from the target can continue to use optimization.
>
> A new migration capability called skip-balloon is introduced. The user can
> disable the capability in cases where user does not expect much benefit or in
> case the migration is from an older version.
>
> During live migration setup the optimization can be set to disabled state if
> . no virtio-balloon device is initialized.
> . skip-balloon migration capability is disabled.
> . If the guest virtio-balloon driver has not set 
> VIRTIO_BALLOON_F_MUST_TELL_HOST
>   flag. Which means the guest may start using a ram pages freed by guest 
> balloon
>   driver, even before the host/qemu is aware of it. In such case, the
>   optimization is disabled so that the ram pages that are being used by the
>   guest will continue to be scanned and migrated.
>
> Balloon bitmap ramblock size is set to zero if the optimization is disabled,
> to avoid overhead of migrating the bitmap. If the bitmap is not migrated to
> the target, the destination starts with a fresh bitmap and tracks the
> ballooning operation thereafter.
>
> Jitendra Kolhe (4):
>   balloon: maintain bitmap for pages released by guest balloon driver.
>   balloon: add balloon bitmap migration capability and setup bitmap
>     migration      status.
>   balloon: reset balloon bitmap ramblock size on source and target.
>   migration: skip scanning and migrating ram pages released by
>     virtio-balloon     driver.
>
> Changed in v2:
>  - Resolved compilation issue for qemu-user binaries in exec.c
>  - Localize balloon bitmap test to save_zero_page().
>  - Updated version string for newly added migration capability to 2.7.
>  - Made minor modifications to patch commit text.
>
> Changed in v3:
>  - Add balloon bitmap to RAMBlock.
>  - Resolve bitmap offset calculation by translating host addr back to a
>    RAMBlock and ram_addr
>  - Add balloon bitmap support for case if TARGET_PAGE_BITS
>    > VIRTIO_BALLOON_PFN_SHIFT.
>  - Remove dependency of skip-balloon migration capability on postcopy
>    migration.
>  - Disable optimization if the guest balloon driver does not support
>    VIRTIO_BALLOON_F_MUST_TELL_HOST feature.
>  - Split single patch into 4 small patches.
>
>  balloon.c                          | 196 
> ++++++++++++++++++++++++++++++++++++-
>  exec.c                             |   6 ++
>  hw/virtio/virtio-balloon.c         |  42 +++++++-
>  include/hw/virtio/virtio-balloon.h |   1 +
>  include/migration/migration.h      |   1 +
>  include/sysemu/balloon.h           |  13 ++-
>  migration/migration.c              |   9 ++
>  migration/ram.c                    |   9 +-
>  qapi-schema.json                   |   5 +-
>  9 files changed, 276 insertions(+), 6 deletions(-)
>
> --
> 1.8.3.1
>
>