qemu-devel
[Top][All Lists]
Advanced

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[Qemu-devel] [PATCH v3 0/4] migration: skip scanning and migrating ram p


From: Jitendra Kolhe
Subject: [Qemu-devel] [PATCH v3 0/4] migration: skip scanning and migrating ram pages released by virtio-balloon driver.
Date: Wed, 18 May 2016 16:50:40 +0530

While measuring live migration performance for qemu/kvm guest, it was observed
that the qemu doesn’t maintain any intelligence for the guest ram pages released
by the guest balloon driver and treat such pages as any other
normal guest ram pages. This has direct impact on overall migration time for
the guest which has released (ballooned out) memory to the host.

In case of large systems, where we can configure large guests with 1TB and
with considerable amount of memory released by balloon driver to the host,
the migration time gets worse.

The solution proposed below is local to qemu (and does not require any
modification to Linux kernel or any guest driver). We have verified the fix
for large guests =1TB on HPE Superdome X (which can support up to 240 cores
and 12TB of memory).

During live migration, as part of first iteration in ram_save_iterate() ->
ram_find_and_save_block () will try to migrate ram pages even if they are
released by vitrio-balloon driver (balloon inflate). Although these pages
which are returned to the host by virtio-balloon driver are zero pages,
the migration algorithm will still end up scanning the entire page
ram_find_and_save_block() -> ram_save_page()/ram_save_compressed_page() ->
save_zero_page() -> is_zero_range(). We also end-up sending header information
over network for these pages during migration. This adds to the total
migration time.

The solution creates a balloon bitmap ramblock as a part of virtio-balloon
device initialization. The bits in the balloon bitmap represent a guest ram
page of size 1UL << VIRTIO_BALLOON_PFN_SHIFT or 4K. If TARGET_PAGE_BITS <=
VIRTIO_BALLOON_PFN_SHIFT, ram_addr offset for the dirty page which is used by
dirty page bitmap during migration is checked against the balloon bitmap as
is, if the bit is set in the balloon bitmap, the corresponding ram page will be
excluded from scanning and sending header information during migration. In case
TARGET_PAGE_BITS > VIRTIO_BALLOON_PFN_SHIFT for a given dirty page ram_addr,
all sub-pages of 1UL << VIRTIO_BALLOON_PFN_SHIFT size should be ballooned out
to avoid zero page scan and sending header information.

The bitmap represents entire guest ram memory till max configured memory.
Guest ram pages claimed by the virtio-balloon driver will be represented by 1
in the bitmap. Since the bitmap is maintained as a ramblock, it’s migrated to
target as part migration’s ram iterative and ram complete phase. So that
substituent migrations from the target can continue to use optimization.

A new migration capability called skip-balloon is introduced. The user can
disable the capability in cases where user does not expect much benefit or in
case the migration is from an older version.

During live migration setup the optimization can be set to disabled state if
. no virtio-balloon device is initialized.
. skip-balloon migration capability is disabled.
. If the guest virtio-balloon driver has not set VIRTIO_BALLOON_F_MUST_TELL_HOST
  flag. Which means the guest may start using a ram pages freed by guest balloon
  driver, even before the host/qemu is aware of it. In such case, the
  optimization is disabled so that the ram pages that are being used by the
  guest will continue to be scanned and migrated.

Balloon bitmap ramblock size is set to zero if the optimization is disabled,
to avoid overhead of migrating the bitmap. If the bitmap is not migrated to
the target, the destination starts with a fresh bitmap and tracks the
ballooning operation thereafter.

Jitendra Kolhe (4):
  balloon: maintain bitmap for pages released by guest balloon driver.
  balloon: add balloon bitmap migration capability and setup bitmap
    migration      status.
  balloon: reset balloon bitmap ramblock size on source and target.
  migration: skip scanning and migrating ram pages released by
    virtio-balloon     driver.

Changed in v2:
 - Resolved compilation issue for qemu-user binaries in exec.c
 - Localize balloon bitmap test to save_zero_page().
 - Updated version string for newly added migration capability to 2.7.
 - Made minor modifications to patch commit text.

Changed in v3:
 - Add balloon bitmap to RAMBlock.
 - Resolve bitmap offset calculation by translating host addr back to a
   RAMBlock and ram_addr
 - Add balloon bitmap support for case if TARGET_PAGE_BITS 
   > VIRTIO_BALLOON_PFN_SHIFT.
 - Remove dependency of skip-balloon migration capability on postcopy
   migration.
 - Disable optimization if the guest balloon driver does not support
   VIRTIO_BALLOON_F_MUST_TELL_HOST feature.
 - Split single patch into 4 small patches.

 balloon.c                          | 196 ++++++++++++++++++++++++++++++++++++-
 exec.c                             |   6 ++
 hw/virtio/virtio-balloon.c         |  42 +++++++-
 include/hw/virtio/virtio-balloon.h |   1 +
 include/migration/migration.h      |   1 +
 include/sysemu/balloon.h           |  13 ++-
 migration/migration.c              |   9 ++
 migration/ram.c                    |   9 +-
 qapi-schema.json                   |   5 +-
 9 files changed, 276 insertions(+), 6 deletions(-)

-- 
1.8.3.1




reply via email to

[Prev in Thread] Current Thread [Next in Thread]