Re: [Qemu-devel] [PATCH v3 0/4] migration: skip scanning and migrating ram pages released by virtio-balloon driver.

[Jitendra Kolhe]

On 1/5/2017 7:03 AM, Li, Liang Z wrote:
>> Am 23.12.2016 um 03:50 schrieb Li, Liang Z:
>>>> While measuring live migration performance for qemu/kvm guest, it was
>>>> observed that the qemu doesn’t maintain any intelligence for the
>>>> guest ram pages released by the guest balloon driver and treat such
>>>> pages as any other normal guest ram pages. This has direct impact on
>>>> overall migration time for the guest which has released (ballooned out)
>> memory to the host.
>>>>
>>>> In case of large systems, where we can configure large guests with
>>>> 1TB and with considerable amount of memory released by balloon driver
>>>> to the host, the migration time gets worse.
>>>>
>>>> The solution proposed below is local to qemu (and does not require
>>>> any modification to Linux kernel or any guest driver). We have
>>>> verified the fix for large guests =1TB on HPE Superdome X (which can
>>>> support up to 240 cores and 12TB of memory).
>>>>
>>>> During live migration, as part of first iteration in
>>>> ram_save_iterate() -> ram_find_and_save_block () will try to migrate
>>>> ram pages even if they are released by vitrio-balloon driver (balloon
>>>> inflate). Although these pages which are returned to the host by
>>>> virtio-balloon driver are zero pages, the migration algorithm will
>>>> still end up scanning the entire page
>>>> ram_find_and_save_block() ->
>>>> ram_save_page()/ram_save_compressed_page() ->
>>>> save_zero_page() -> is_zero_range(). We also end-up sending header
>>>> information over network for these pages during migration. This adds
>>>> to the total migration time.
>>>>
>>>> The solution creates a balloon bitmap ramblock as a part of
>>>> virtio-balloon device initialization. The bits in the balloon bitmap
>>>> represent a guest ram page of size 1UL << VIRTIO_BALLOON_PFN_SHIFT
>> or
>>>> 4K. If TARGET_PAGE_BITS <= VIRTIO_BALLOON_PFN_SHIFT, ram_addr
>> offset
>>>> for the dirty page which is used by dirty page bitmap during
>>>> migration is checked against the balloon bitmap as is, if the bit is
>>>> set in the balloon bitmap, the corresponding ram page will be
>>>> excluded from scanning and sending header information during
>>>> migration. In case TARGET_PAGE_BITS > VIRTIO_BALLOON_PFN_SHIFT
>> for a
>>>> given dirty page ram_addr, all sub-pages of 1UL <<
>>>> VIRTIO_BALLOON_PFN_SHIFT size should be ballooned out to avoid zero
>> page scan and sending header information.
>>>>
>>>> The bitmap represents entire guest ram memory till max configured
>> memory.
>>>> Guest ram pages claimed by the virtio-balloon driver will be
>>>> represented by 1 in the bitmap. Since the bitmap is maintained as a
>>>> ramblock, it’s migrated to target as part migration’s ram iterative
>>>> and ram complete phase. So that substituent migrations from the target
>> can continue to use optimization.
>>>>
>>>> A new migration capability called skip-balloon is introduced. The
>>>> user can disable the capability in cases where user does not expect
>>>> much benefit or in case the migration is from an older version.
>>>>
>>>> During live migration setup the optimization can be set to disabled state 
>>>> if .
>>>> no virtio-balloon device is initialized.
>>>> . skip-balloon migration capability is disabled.
>>>> . If the guest virtio-balloon driver has not set
>>>> VIRTIO_BALLOON_F_MUST_TELL_HOST
>>>>   flag. Which means the guest may start using a ram pages freed by
>>>> guest balloon
>>>>   driver, even before the host/qemu is aware of it. In such case, the
>>>>   optimization is disabled so that the ram pages that are being used by the
>>>>   guest will continue to be scanned and migrated.
>>>>
>>>> Balloon bitmap ramblock size is set to zero if the optimization is
>>>> disabled, to avoid overhead of migrating the bitmap. If the bitmap is
>>>> not migrated to the target, the destination starts with a fresh
>>>> bitmap and tracks the ballooning operation thereafter.
>>>>
>>>
>>> I have a better way to get rid of the bitmap.
>>> We should not maintain the inflating pages in the bitmap, instead, we
>>> can get them from the guest if it's needed, just like what we did for
>>> the guest's unused pages. Then we can combine the inflating page info
>>> with the unused page info together, and skip them during live migration.
>>>
>>
>> If we want to actually host enforce (disallow) access to inflated pages, 
>> having
> 
> Is that a new feature?
> 
>> such a bitmap in QEMU is required. Getting them from the guest doesn't
>> make sense in that context anymore.
> 
> Even a bitmap is required, we should avoid to send it in the stop & copy 
> stage.
> 

How about using both ways? If the guest is capable to share ballooned out page
information using unused page framework use it. If the guest is not capable we
maintain a bitmap in qemu. This way we can use advantage of both approaches. 
1. In case the guest is capable - we avoid bitmap send in stop and copy stage.
2. In case the guest is not capable - host can still disallow migration of 
ballooned 
out pages.
But since this approach may add a bit of code churn, we recommend to go with 
maintaining 
the bitmap in qemu at first phase (existing implementation) and then enhance it 
later 
to use unused page approach (to query guest) based on the guest capability. 

Thanks,
- Jitendra

> Thanks!
> Liang
>>
>> Thanks,
>> David
>>