Re: [Qemu-devel] [RFC] Incremental live backup with in memory dirty bitmap

[Stefan Hajnoczi]

On Mon, Nov 25, 2013 at 05:59:12PM +0800, Fam Zheng wrote:
> This is an idea about allowing online incremental backup of block
> device, with drive-backup and (proposed here) in-memory block dirty
> bitmap:
> 
> 1. We enable a dirty bitmap on a block device, at the start point of
> write tracking:
> 
>   (QMP) dirty-bitmap-add device=foo name=bitmap0 granularity=4k
> 
> Since now, all the writes to "foo" will mark dirty bits on it.
> 
> 2.1 Later, when the user wants to export all the dirty data, one
> possible approach is using image fleecing:
> 
>   # step a. prepare the image fleecing target
>   (QMP) blockdev-add backing=foo file.filename=backup.qcow2
> id=backup0 if=none driver=qcow2
> 
>   # step b. stop the dirty bitmap and start backup job. These two
> steps need to be in a transaction, so the exported data matches the
> exported dirty bitmap
>   (QMP) transaction:
>             dirty-bitmap-stop device=foo name=bitmap0
>             blockdev-backup device=foo sync=none target=backup0
> 
>   # step c. export the snapshot:
>   (QMP) nbd-server-add device=backup0
> 
>   # step d. export the dirty bitmap as well, so the management
> application can access it and decide what sectors to backup
>   (QMP) dirty-bitmap-nbd-add device=foo name=bitmap0
> 
> Now the management application can connect to both dirty bitmap
> target and image fleecing target, and read whatever data it is
> interested in.
> 
> The tricky part is to export the dirty bitmap through NBD, but it
> may also be useful for persistent the in-memory dirty bitmap too.
> 
> 2.2 Alternatively, we can avoid exporting the dirty bitmap through
> NBD, by adding another sync mode "dirty" to drive-backup, and change
> step b to:
> 
>   # step b. stop the dirty bitmap and start backup job with the
> dirty bitmap
>   (QMP) transaction:
>             dirty-bitmap-stop device=foo name=bitmap0
>             blockdev-backup device=foo sync=dirty bitmap=bitmap0
> target=backup0
> 
> With the "dirty" sync mode, drive-backup/blockdev-backup takes a
> dirty bitmap name of the device and writes only dirty data to
> target.
> 
> Note that NBD is not necessary for backup, we can use "drive-backup"
> and omit step c as well. Also, the target could be an NBD server, in
> which case we will directly send new data to a remote NBD target.
> This part is fully flexible.
> 
> Summary: what we are missing are the dirty bitmap interfaces to
> add/stop/remove/query/export it and it's support of transaction,
> depends on how we want to use it. E.g. for continuity of the
> incremental tracking, the transaction could be:
> 
>   (QMP) transaction:
>             dirty-bitmap-stop device=foo name=bitmap0
>             dirty-bitmap-add device=foo name=bitmap1   # Start a new
> bitmap atomically
>             blockdev-backup device=foo sync=dirty bitmap=bitmap0
> target=backup0
> 
> And the last command is to free the no longer used dirty bitmap:
> 
>   (QMP) dirty-bitmap-remove device=foo name=bitmap0
> 
> So, do these sound good/useful at all? Any comment is welcome!

This sounds like a good idea.  Incremental backup has been discussed
periodically and we're in a good position to actually implement it now.

The dirty bitmap object's lifecycle needs to be fully thought through,
including how to make the bitmap persistent.  I think a reasonable
approach is flushing the bitmap to disk when QEMU shuts down.  The
bitmap file should be marked "in use" by dirty-bitmap-start so the
contents are not trusted after a power failure/crash.

I like the blockdev-backup sync=dirty mode which can be used to stream
dirty blocks to an external NBD server.

The minimal feature set that covers persistent dirty bitmaps is:
1. dirty-bitmap-add/dirty-bitmap-remove
2. blockdev-backup sync=dirty

The user would not be able to query the bitmap itself but they could
receive a copy of changed blocks.

Eric: any thoughts on persistent dirty bitmap APIs?

Stefan