[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[Qemu-devel] [RFC V5 00/62] QCOW2 deduplication
From: |
Benoît Canet |
Subject: |
[Qemu-devel] [RFC V5 00/62] QCOW2 deduplication |
Date: |
Wed, 16 Jan 2013 16:47:39 +0100 |
This 3 step patchset implements deduplication in QCOW2.
First patchset create the core infrastructure for deduplication and enable it
in QCOW2 image.
It ends at "qcow2: Enable the deduplication feature."
Second patchset implements some metrics in QMP.
It ends at "qapi: Return virtual block device deduplication metrics in QMP"
Third patchset implements asynchronous deduplication.
It's a work in progress patchset that is included in this post so reviewers
can have a grasp of where the feature is heading.
One can compile and install https://github.com/wernerd/Skein3Fish and use the
--enable-skein-dedup configure option in order to use the faster skein HASH.
Images must be created with "-o dedup=[skein|sha256]" in order to activate the
deduplication in the image.
Deduplication is now fast enough to be usable.
Nice side effect is that duplicated writes are faster than native QCOW2:
v5:
Move qemu-io-test dedup patch [Eric]
Reserve some room at the end of the QCOW header extensions. [Eric]
Fix the specification. [Eric]
Now overflow deduplication refcount at 2^16/2 [Stefan]
Implements metrics.
Implement asynchronous deduplication.
Increase L2 table size and deduplication block hash size.
Random cleanups
v4: Fix and complete qcow2 spec [Stefan]
Hash the hash_algo field in the header extension [Stefan]
Fix qcow2 spec [Eric]
Remove pointer to hash and simplify hash memory management [Stefan]
Rename and move qcow2_read_cluster_data to qcow2.c [Stefan]
Document lock dropping behaviour of the previous function [Stefan]
cleanup qcow2_dedup_read_missing_cluster_data [Stefan]
rename *_offset to *_sect [Stefan]
add a ./configure check for ssl [Stefan]
Replace openssl by gnutls [Stefan]
Implement Skein hashes
Rewrite pretty every qcow2-dedup.c commits after Add
qcow2_dedup_read_missing_and_concatenate to simplify the code
Use 64KB deduplication hash block to reduce allocation flushes
Use 64KB l2 tables to reduce allocation flushes [breaks compatibility]
Use lazy refcounts to avoid qcow2_cache_set_dependency loops resultings
in frequent caches flushes
Do not create and load dedup RAM structures when bdrs->read_only is true
v3: make it work barely
replace kernel red black trees by gtree.
Benoît Canet (62):
qcow2: Add deduplication to the qcow2 specification.
qcow2: Add deduplication structures and fields.
qcow2: Add qcow2_dedup_read_missing_and_concatenate
qcow2: Make update_refcount public.
qcow2: Create a way to link to l2 tables when deduplicating.
qcow2: Add qcow2_dedup and related functions
qcow2: Add qcow2_dedup_store_new_hashes.
qcow2: Implement qcow2_compute_cluster_hash.
qcow2: Extract qcow2_dedup_grow_table
qcow2: Add qcow2_dedup_grow_table and use it.
qcow2: Makes qcow2_alloc_cluster_link_l2 mark to deduplicate
clusters.
qcow2: make the deduplication forget a cluster hash when a cluster is
to dedupe
qcow2: Create qcow2_is_cluster_to_dedup.
qcow2: Load and save deduplication table header extension.
qcow2: Extract qcow2_do_table_init.
qcow2-cache: Allow to choose table size at creation.
qcow2: Extract qcow2_add_feature and qcow2_remove_feature.
block: Add qemu-img dedup create option.
qcow2: Add a deduplication boolean to update_refcount.
qcow2: Drop hash for a given cluster when dedup makes refcount >
2^16/2.
qcow2: Remove hash when cluster is deleted.
qcow2: Add qcow2_dedup_is_running to probe if dedup is running.
qcow2: Integrate deduplication in qcow2_co_writev loop.
qcow2: Serialize write requests when deduplication is activated.
qcow2: Add verification of dedup table.
qcow2: Adapt checking of QCOW_OFLAG_COPIED for dedup.
qcow2: Add check_dedup_l2 in order to check l2 of dedup table.
qcow2: Do not overwrite existing entries with QCOW_OFLAG_COPIED.
qcow2: Integrate SKEIN hash algorithm in deduplication.
qcow2: Add lazy refcounts to deduplication to prevent
qcow2_cache_set_dependency loops
qcow2: Use large L2 table for deduplication.
qcow: Set large dedup hash block size.
qemu-iotests: Filter dedup=on/off so existing tests don't break.
qcow2: Add qcow2_dedup_init and qcow2_dedup_close.
qcow2: Add qcow2_co_dedup_resume to restart deduplication.
qcow2: Enable the deduplication feature.
qcow2: Add deduplication metrics structures.
qcow2: Initialize deduplication metrics.
qcow2: Collect unaligned writes missing data reads metric.
qcow2: Collect deduplicated cluster metric.
qcow2: Collect undeduplicated cluster metric.
qcow2: Count QCowHashNode creation metrics.
qcow2: Count QCowHashNode removal from tree for metrics.
qcow2: Count cluster deleted metric
qcow2: Count deduplication refcount overflow metric.
qapi: Add support for deduplication infos in qapi-schema.json.
block: Add deduplication metrics to BlockDriverInfo.
qcow2: Add qcow2_dedup_update_metrics to compute dedup RAM usage.
qcow2: returns deduplication metrics and status via bdrv_get_info()
qapi: Return virtual block device deduplication metrics in QMP
block: Add BlockDriver function prototype to pause and resume
deduplication.
qcow2: Add code to deduplicate cluster flagged with
QCOW_OFLAG_TO_DEDUP.
block: Add bdrv_has_dedup.
block: Add bdrv_is_dedup_running.
block: Add bdrv_resume_dedup.
block: Add bdrv_pause_dedup.
qcow2: Add qcow2_pause_dedup.
qcow2: Add qcow2_resume_dedup.
qcow2: Make dedup status persists.
qerror: Add QERR_DEVICE_NOT_DEDUPLICATED.
qmp: Add block-pause-dedup.
qmp: Add block_resume_dedup.
block.c | 108 +++
block/Makefile.objs | 1 +
block/qcow2-cache.c | 12 +-
block/qcow2-cluster.c | 182 ++++--
block/qcow2-dedup.c | 1492 ++++++++++++++++++++++++++++++++++++++++++
block/qcow2-refcount.c | 175 +++--
block/qcow2.c | 378 +++++++++--
block/qcow2.h | 149 ++++-
blockdev.c | 36 +
configure | 55 ++
docs/specs/qcow2.txt | 104 ++-
include/block/block.h | 18 +
include/block/block_int.h | 5 +
include/qapi/qmp/qerror.h | 3 +
qapi-schema.json | 76 ++-
qmp-commands.hx | 46 ++
tests/qemu-iotests/common.rc | 3 +-
17 files changed, 2708 insertions(+), 135 deletions(-)
create mode 100644 block/qcow2-dedup.c
--
1.7.10.4
- [Qemu-devel] [RFC V5 00/62] QCOW2 deduplication,
Benoît Canet <=
- [Qemu-devel] [RFC V5 02/62] qcow2: Add deduplication structures and fields., Benoît Canet, 2013/01/16
- [Qemu-devel] [RFC V5 01/62] qcow2: Add deduplication to the qcow2 specification., Benoît Canet, 2013/01/16
- [Qemu-devel] [RFC V5 04/62] qcow2: Make update_refcount public., Benoît Canet, 2013/01/16
- [Qemu-devel] [RFC V5 03/62] qcow2: Add qcow2_de dup_read_missing_and_concatenate, Benoît Canet, 2013/01/16
- [Qemu-devel] [RFC V5 05/62] qcow2: Create a way to link to l2 tables when deduplicating., Benoît Canet, 2013/01/16
- [Qemu-devel] [RFC V5 08/62] qcow2: Implement qcow2_compute_cluster_hash., Benoît Canet, 2013/01/16
- [Qemu-devel] [RFC V5 07/62] qcow2: Add qcow2_dedup_store_new_hashes., Benoît Canet, 2013/01/16
- [Qemu-devel] [RFC V5 06/62] qcow2: Add qcow2_dedup and related functions, Benoît Canet, 2013/01/16