Re: [Qemu-devel] [RFC V3 00/24] QCOW2 deduplication

[Benoît Canet]

Hi Stefan,

I have a few questions

1) overlapping sequential sub-cluster writes

The current code pass most of the tests and behave well with a 4KB cluster sized
ext3 volume on the deduplicated image.

But less than cluster size sequentials writes are troublesome.
They fail with xfstest.
The problem is that the lock is released twice so that coherency is not
garanteed when two sub cluster size write are done on the same area.
(a deduplication attempt is done while the first write is yet not on disk)

My understanding is that a wait_for_overlapping_cluster_write function called
before the writev loop in order to serialize such writes would solve the 
problem.
What do you this of this idea ?

2) Internal snapshot
I don't fully understand if the current deduplication implementation is
compatible with internal snapshots. If not could it be done on a latter
patchset ?

Benoît

> Le Monday 26 Nov 2012 à 14:04:59 (+0100), Benoît Canet a écrit :
> This patchset is the first working version of the QCOW2 deduplication.
> 
> Images must be created with "-o dedup=on" in order to activate the
> deduplication in the image.
> 
> 
> Since v2: make it work barely
>           replace kernel red black trees by gtree.
> 
> Benoît Canet (24):
>   qcow2: Add deduplication to the qcow2 specification.
>   qcow2: Add deduplication structures and fields.
>   qcow2: Add qcow2_dedup_read_missing_and_concatenate
>   qcow2: Make update_cluster_refcount public.
>   qcow2: Create a way to link to l2 tables in dedup.
>   qcow2: Add qcow2_dedup and related functions.
>   qcow2: Add qcow2_dedup_write_new_hashes.
>   qcow2: Implement qcow2_compute_cluster_hash.
>   qcow2: Extract qcow2_dedup_grow_table
>   qcow2: create function to load deduplication hashes at startup.
>   qcow2: Load and save deduplication table header extension.
>   qcow2: Extract qcow2_do_table_init.
>   qcow2: Add qcow2_dedup_init and qcow2_dedup_close.
>   qcow2: Extract qcow2_add_feature and qcow2_remove_feature.
>   block: Add dedup image create option.
>   qcow2: Allow creation of images using deduplication.
>   qcow2: Behave correctly when refcount reach 0 or 2^16.
>   qcow2: Integrate deduplication in qcow2_co_writev loop.
>   qcow2: Add verification of dedup table.
>   qcow2: Adapt checking of QCOW_OFLAG_COPIED for dedup.
>   qcow2: Add check_dedup_l2 in order to check l2 of dedup table.
>   qcow2: Do not overwrite existing entries with QCOW_OFLAG_COPIED.
>   qcow2: init and cleanup deduplication.
>   qemu-iotests: Filter dedup=on/off so existing tests don't break.
> 
>  Makefile                     |    3 +
>  Makefile.target              |    2 +-
>  block/Makefile.objs          |    1 +
>  block/qcow2-cluster.c        |  115 ++++--
>  block/qcow2-dedup.c          |  914 
> ++++++++++++++++++++++++++++++++++++++++++
>  block/qcow2-refcount.c       |  154 +++++--
>  block/qcow2.c                |  267 ++++++++++--
>  block/qcow2.h                |   89 +++-
>  block_int.h                  |    1 +
>  docs/specs/qcow2.txt         |   33 +-
>  tests/qemu-iotests/common.rc |    3 +-
>  11 files changed, 1480 insertions(+), 102 deletions(-)
>  create mode 100644 block/qcow2-dedup.c
> 
> -- 
> 1.7.10.4
>