Re: [Qemu-devel] [RFC V8 01/24] qcow2: Add journal specification.

[Stefan Hajnoczi]

On Tue, Jul 02, 2013 at 11:23:56PM +0200, Benoît Canet wrote:
> > > +QCOW2 can use one or more instance of a metadata journal.
> > 
> > s/instance/instances/
> > 
> > Is there a reason to use multiple journals rather than a single journal
> > for all entry types?  The single journal area avoids seeks.
> 
> Here are the main reason for this:
> 
> For the deduplication some patterns like cycles of insertion/deletion could
> leave the hash table almost empty while filling the journal.
> 
> If the journal is full and the hash table is empty a packing operation is
> started.
> 
> Basically a new journal is created and only the entry presents in the hash 
> table
> are reinserted.
> 
> This is why I want to keep the deduplication journal appart from regular qcow2
> journal: to avoid interferences between a pack operation and regular qcow2
> journal entries.
> 
> The other thing is that freezing the log store would need a replay of regular
> qcow2 entries as it trigger a reset of the journal.
> 
> Also since deduplication will not work on spinning disk I discarded the seek
> time factor.
> 
> Maybe commiting the dedupe journal by erase block sized chunk would be a good
> idea to reduce random writes to the SSD.
> 
> The additional reason for having multiple journals is that the SILT paper
> propose a mode where prefix of the hash is used to dispatch insertions in
> multiples store and it easier to do with multiple journals.

It sounds like the journal is more than just a data integrity mechanism.
It's an integral part of your dedup algorithm and you plan to carefully
manage it while rebuilding some of the other dedup data structures.

Does this mean the journal forms the first-stage data structure for
deduplication?  Dedup records will accumulate in the journal until it
becomes time to convert them in bulk into a more compact representation?

When I read this specification I was thinking of a journal purely for
logging operations.  You could use a commit record to mark previous
records applied.  Upon startup, qcow2 would inspect uncommitted records
and deal with them.

We just need to figure out how to define a good interface so that the
journal can be used in a general way but also for dedup's specific
needs.

Stefan