Re: [Qemu-devel] [PATCH v3 0/8] block: drive-backup live backup command

[Stefan Hajnoczi]

On Thu, May 23, 2013 at 08:11:42AM +0000, Dietmar Maurer wrote:
> > > I also consider it safer, because you make sure the data exists (using 
> > > hash keys
> > like SHA1).
> > >
> > > I am unsure how you can check if a dirty bitmap contains errors, or is 
> > > out of
> > date?
> > >
> > > Also, you can compare arbitrary Merkle trees, whereas a dirty bitmap is 
> > > always
> > related to a single image.
> > > (consider the user remove the latest backup from the backup target).
> > 
> > One disadvantage of Merkle trees is that the client becomes stateful - the 
> > client
> > needs to store its own Merkle tree and this requires fancier client-side 
> > code.
> 
> What 'client' do you talk about here?

A backup application, for example.  Previously it could simply use
api.getDirtyBlocks() -> [Sector] and it would translate into a single
QMP API call.

Now a Merkle tree needs to be stored on the client side and synced with
the server.  The client-side library becomes more complex.

> But sure, the code gets more complex, and needs considerable amount of RAM
> to store the hash keys .
>  
> > It is also more expensive to update hashes than a dirty bitmap.  Not 
> > because you
> > need to hash data but because a small write (e.g. 1 sector) requires that 
> > you
> > read the surrounding sectors to recompute a hash for the cluster.  
> > Therefore you
> > can expect worse guest I/O performance than with a dirty bitmap.
> 
> There is no need to update any hash - You only need to do that on backup - in 
> fact, all
> those things can be done by the backup driver.

The problem is that if you leave hash calculation until backup time then
you need to read in the entire disk image (100s of GB) from disk.  That
is slow and drains I/O resources.

Maybe the best approach is to maintain a dirty bitmap while the guest is
running, which is fairly cheap.  Then you can use the dirty bitmap to
only hash modified clusters when building the Merkle tree - this avoids
reading the entire disk image.

> > I still think it's a cool idea.  Making it work well will require a lot 
> > more effort than
> > a dirty bitmap.
> 
> How do you re-generate a dirty bitmap after a server crash?

The dirty bitmap is invalid after crash.  A full backup is required, all
clusters are considered dirty.

The simplest way to implement this is to mark the persistent bitmap
"invalid" upon the first guest write.  When QEMU is terminated cleanly,
flush all dirty bitmap updates to disk and then mark the file "valid"
again.  If QEMU finds the file is "invalid" on startup, start from
scratch.

Stefan