Re: [Qemu-devel] [PATCH v5 00/12] Dirty bitmaps migration

[John Snow]

On 05/21/2015 09:57 AM, Denis V. Lunev wrote:
> On 21/05/15 16:51, Vladimir Sementsov-Ogievskiy wrote:
>> Hi all.
>>
>> Hmm. There is an interesting suggestion from Denis Lunev (in CC) about
>> how to drop meta bitmaps and make things easer.
>>
>> method:
>>
>> > start migration
>> disk and memory are migrated, but not dirty bitmaps.
>> > stop vm
>> create all necessary bitmaps in destination vm (empty, but with same
>> names and granularities and enabled flag)
>> > start destination vm
>> empty bitmaps are tracking now
>> > start migrating dirty bitmaps. merge them to corresponding bitmaps
>> in destination
>> while bitmaps are migrating, they should be in some kind of
>> 'inconsistent' state.
>> so, we can't start backup or other migration while bitmaps are
>> migrating, but vm is already _running_ on destination.
>>
>> what do you think about it?
>>
> 
> the description is a bit incorrect
> 
> - start migration process, perform memory and disk migration
>    as usual. VM is still executed at source
> - start VM on target. VM on source should be on pause as usual,
>    do not finish migration process. Running VM on target "writes"
>    normally setting dirty bits as usual
> - copy active dirty bitmaps from source to target. This is safe
>    as VM on source is not running
> - "OR" copied bitmaps with ones running on target
> - finish migration process (stop source VM).
> 
> Downtime will not be increased due to dirty bitmaps with this
> approach, migration process is very simple - plain data copy.
> 
> Regards,
>     Den
> 

I was actually just discussing the live migration approach a little bit
ago with Stefan, trying to decide on the "right" packet format (The only
two patches I haven't ACKed yet are ones in which we need to choose a
send size) and we decided that 1KiB chunk sends would be appropriate for
live migration.

I think I'm okay with that method, but obviously this approach outlined
here would also work very well and would avoid meta bitmaps, chunk
sizes, migration tuning, convergence questions, etc etc etc.

You'd need to add a new status to the bitmap on the target (maybe
"INCOMPLETE" or "MIGRATING") that prevents it from being used for a
backup operation without preventing it from recording new writes.

My only concern is how easy it will be to work this into the migration
workflow.

It would require some sort of "post-migration" ternary phase, I suppose,
for devices/data that can be transferred after the VM starts -- and I
suspect we'll be the only use of that phase for now.

David, what are your thoughts, here? Would you prefer Vladimir and I
push forward on the live migration approach, or add a new post-hoc
phase? This approach might be simpler on the block layer, but I would be
rather upset if he scrapped his entire series for the second time for
another approach that also didn't get accepted.

--js

>> On 13.05.2015 18:29, Vladimir Sementsov-Ogievskiy wrote:
>>> These patches provide dirty bitmap migration feature. Only named dirty
>>> bitmaps are to be migrated. Migration may be enabled using migration
>>> capabilities.
>>>
>>> v5:
>>>      - rebase on master
>>>      - drop [PATCH RFC v4 10/13] iotests: add event_wait to VM class
>>>      - remove rfc, as incremental backup series by John Snow are in
>>>        upstream
>>>
>>> v4 significant changes:
>>>   0001: tiny bugfix: out[i] -> out[i-start], same for 'in'
>>>   0007: set chunk size to 1kb, disable live iteration for
>>>         migrating data < 1mb size.
>>>
>>>   tests: only one with md5 sum is here. used function event_wait
>>>          by John Snow. (I hope, you don't mind me just adding this
>>>          function with your 'Signed-off-by')
>>>
>>>   rfc: This patch set is based on v13 of
>>>        "block: incremental backup series" by John Snow, which are
>>>        not pushed yet.
>>>
>>> v3:
>>>   based on v13 of "block: incremental backup series" by John Snow.
>>>
>>>   changes from v2:
>>>   removed patch for adding dirty parameter (migration capablities used
>>>   instead).
>>>     0001: printf's dropped, qapi used
>>>   0002: part0 -> zeroes
>>>   0003: part0 -> zeroes
>>>   0005: dirty_dirty -> meta
>>>         add comments about meta bitmap
>>>           0006: the format is changed, nodes used instead of devices.
>>>
>>>   other patches are new.
>>>
>>>   rfc: there are two tests. They are the same but using different
>>>   interfaces: md5 checksum of the bitmap last layer in query-block or
>>>   separate query-block-dirty-bitmap with dirty bitmap regions.
>>>   The second form is more appropriate for debugging, the first is more
>>>   appropriate for simple regression control. Which should go to
>>>   upstream?
>>>
>>> v2:
>>>   1. bug-fixes, that are already in upstream, and renaming of function
>>>   bdrv_reset_dirty_bitmap (which is already in Snow's series) are
>>>   dropped
>>>   2. bitmap store/restore: the concept renamed to serialization, added
>>>   function hbitmap_deserialize_part0, to not transfer zero blocks
>>>   3. migration dirty parameter: added description comment
>>>   4. Other patches are new.
>>>
>>> v2.rfc:
>>> Actually, in this version of the series I'm trying not use
>>> migration/block.c at all. Instead a separate migration unit is added
>>> in the new file migration/dirty-bitmap.c. Now bitmaps are migrated
>>> like blocks in block migration, they have their "dirty-dirty" bitmaps,
>>> for tracking set/unset changes during migration.
>>>
>>> The advantages are:
>>>    - no complications of migration/block.c
>>>    - separate dirty-dirty bitmaps provide handling of "unset's"
>>>    - more effective meta-data/data ratio - no tiny bitmap-blocks.
>>>
>>>
>>>
>>> v1:
>>> These patches provide dirty bitmap migration feature. Only named dirty
>>> bitmaps are to be migrated. Migration is made as a part of block
>>> migration in block-migration.c.
>>>
>>> Dirty bitmap migration may be enabled by "dirty" parameter for qmp
>>> migrate
>>> command. If "blk" and "inc" parameters are false when "dirty" is true
>>> block migration is actually skipped: no allocatoions, no bdrv_read's,
>>> no bdrv_write's, only bitmaps are migrated.
>>>
>>> The patch set includes two my previous bug fixes, which are necessary
>>> for it. The patch set is based on Incremental backup series by John
>>> Snow.
>>> Vladimir Sementsov-Ogievskiy (12):
>>>    hbitmap: serialization
>>>    block: BdrvDirtyBitmap serialization interface
>>>    block: tiny refactoring: minimize hbitmap_(set/reset) usage
>>>    block: add meta bitmaps
>>>    block: add bdrv_next_dirty_bitmap()
>>>    qapi: add dirty-bitmaps migration capability
>>>    migration: add migration/block-dirty-bitmap.c
>>>    iotests: maintain several vms in test
>>>    iotests: add add_incoming_migration to VM class
>>>    qapi: add md5 checksum of last dirty bitmap level to query-block
>>>    iotests: add dirty bitmap migration test
>>>    migration/qemu-file: make functions qemu_(get/put)_string public
>>>
>>>   block.c                        |  98 +++++-
>>>   include/block/block.h          |  22 ++
>>>   include/migration/block.h      |   1 +
>>>   include/migration/migration.h  |   1 +
>>>   include/migration/qemu-file.h  |  17 +
>>>   include/qemu/hbitmap.h         |  67 ++++
>>>   migration/Makefile.objs        |   2 +-
>>>   migration/block-dirty-bitmap.c | 693
>>> +++++++++++++++++++++++++++++++++++++++++
>>>   migration/migration.c          |   9 +
>>>   migration/qemu-file.c          |  18 ++
>>>   qapi-schema.json               |   4 +-
>>>   qapi/block-core.json           |   4 +-
>>>   tests/qemu-iotests/117         |  84 +++++
>>>   tests/qemu-iotests/117.out     |   5 +
>>>   tests/qemu-iotests/group       |   1 +
>>>   tests/qemu-iotests/iotests.py  |  14 +-
>>>   util/hbitmap.c                 | 106 +++++++
>>>   vl.c                           |   1 +
>>>   18 files changed, 1137 insertions(+), 10 deletions(-)
>>>   create mode 100644 migration/block-dirty-bitmap.c
>>>   create mode 100755 tests/qemu-iotests/117
>>>   create mode 100644 tests/qemu-iotests/117.out
>>>
>>
>>
>