[Qemu-devel] RFC: incremental backups: qmp-block-dirty-bitmap-diff

[John Snow]

The thought was raised that it might be useful to create a "diff"
command for bitmaps that allow us to populate a dirty bitmap based on
the difference between a currently loaded disk image and some
point-in-time stored on disk (an earlier full backup, incremental
backup, etc).

I originally voiced a desire for this command in response to how to
recover from a failed post-copy bitmap migration (Talks ongoing for if
we'll pursue that, but let's not worry about that here.)

Another usage (apart from disaster recovery where we -- for whatever
reason -- lost our bitmap or actually even lost an intermediate
differential backup) for this kind of functionality would be to *start*
a new incremental backup chain in a post-hoc fashion -- after we already
made a full backup, but neglected to create a bitmap atomically.

There are some pros and cons here; I wanted to get a general "Yes, that
sounds good" or a "No way, don't bother" before I sunk some hours into
it, at Stefan's suggestion.


Cons:
- Added maintenance burden of yet-another-blockjob that does a fairly
esoteric operation

- Catering to user carelessness. "You should have thought of that earlier!"

- Reading an entire drive could take quite a long time -- Especially if
both the source and the target are through network storage.

- Reading an entire drive to populate a bitmap with the understanding
that an incremental backup is soon to follow is inefficient if the drive
is more than just a little dirty: it may have been quicker to just
create a new full backup and bitmap.

- Not strictly unlocking new functionality in the normative case. (We do
add 'bitmap rebuilding after a freak accident', but this is not the
normative case.)


Pros:
- Since bitmaps (and persistent bitmaps) are off by default, it's nice
to allow users to enable the feature post-hoc. The chance for a
misconfiguration here seems reasonably good, especially considering the
feature hasn't really hit all of the stable distros yet.

- Increases the flexibility overall of how to begin using the
incremental backups feature.

- Allows us to sync bitmaps against images created by external tools if
needed or desired by external backup appliances.

- If the image isn't that much different from the last good backup,
reading the entire drive and writing out only a few changed blocks is
likely to be faster than creating a new full backup, considering reads
tend to be faster than writes -- reading the whole drive will be faster
than writing a new full backup in many cases.

- The fact that a diff+incremental backup may be slower in some cases
than just simply creating a new full backup isn't a good rationale for
why we can't offer this tool for the cases in which it may outperform.

- Read of the last-known-good backup can be done in the background as a
blockjob without impeding VM operation. Writes that occur during this
process can be tracked in a temporary bitmap and merged after the diff
completes.