[Qemu-devel] [PATCH v3 00/12] qcow2: Add new overlap check functions

[Max Reitz]

As has been requested, this series adds new overlap check functions to
the qcow2 code. My local branch is called "qcow2-improved-overlap-v1",
but I am not so sure whether it is actually an improvement; that is left
for you to decide, dear reviewers.

See patch 1 for an explanation of why this series exists and what it
does. Patch 1 is basically the core of this series, the rest just
employs the functions introduced there.

In a later patch, we may want to change the meaning of the "constant"
overlap checking option to mean the same as "cached", which is
everything except for inactive L2 tables. This series does make
checking for overlaps with inactive L2 tables at runtime just as cheap
as everything else (constant time plus caching), but using these checks
means qemu has to read all the snapshot L1 tables when opening a qcow2
file. This does not take long, of course, but it does result in a bit of
overhead so I did not want to enable it by default.

I think just enabling all overlap checks by default after this series
should be fine and useful, though.


For benchmarks, please see my cover letter for v2:
http://lists.nongnu.org/archive/html/qemu-devel/2014-11/msg03430.html

I'll just quote the (fixed) tl;dr here:

* CPU usage at runtime decreased by 15000 to 27500 percent on
  overlap-check-heavy tasks
* No additional performance problems at loading time (in theory has the
  same runtime complexity as a single overlap check right now; in
  practice I could not find any problems)
* Decent RAM usage (170 kB for a 1 TB image with 64 kB clusters; 170 MB
  for a 1 PB image etc. pp.)


v3:
- Patch 1:
  - s/few memory/little memory/ [Eric]
  - s/2014/2015/ [Eric]
  - s/of a or a/of or a/ [Eric]
  - Use nb_clusters_minus_one instead of nb_clusters, because
    nb_clusters == 0 is invalid [Eric]
  - Drop cached_windows_index [Stefan]
  - Reset current_nb_clusters to 1 when a new range starts in the
    algorithm which rebuilds the fragment list from the bitmap
    representation
- Patch 3: Added a TODO comment that the user should be able to override
  the default cache size limit
- Patch 8: As the patch "qcow2: Buffer L1 table in snapshot refcount
  update" has been dropped, this patch was changed to no longer rely on
  it
- Patch 11: Replaced the size limit for a snapshot's L1 table by
  QCOW_MAX_L1_SIZE (was: INT_MAX / sizeof(uint64_t)) [Eric]


git-backport-diff against v2:

Key:
[----] : patches are identical
[####] : number of functional differences between upstream/downstream patch
[down] : patch is downstream-only
The flags [FC] indicate (F)unctional and (C)ontextual differences, respectively

001/12:[0026] [FC] 'qcow2: Add new overlap check functions'
002/12:[----] [--] 'qcow2: Pull up overlap check option evaluation'
003/12:[0001] [FC] 'qcow2: Create metadata list'
004/12:[----] [--] 'qcow2/overlaps: Protect image header'
005/12:[----] [--] 'qcow2/overlaps: Protect refcount table'
006/12:[----] [--] 'qcow2/overlaps: Protect refcount blocks'
007/12:[----] [--] 'qcow2/overlaps: Protect active L1 table'
008/12:[0002] [FC] 'qcow2/overlaps: Protect active L2 tables'
009/12:[----] [--] 'qcow2/overlaps: Protect snapshot table'
010/12:[----] [--] 'qcow2/overlaps: Protect inactive L1 tables'
011/12:[0002] [FC] 'qcow2/overlaps: Protect inactive L2 tables'
012/12:[----] [--] 'qcow2: Use new metadata overlap check function'


Max Reitz (12):
  qcow2: Add new overlap check functions
  qcow2: Pull up overlap check option evaluation
  qcow2: Create metadata list
  qcow2/overlaps: Protect image header
  qcow2/overlaps: Protect refcount table
  qcow2/overlaps: Protect refcount blocks
  qcow2/overlaps: Protect active L1 table
  qcow2/overlaps: Protect active L2 tables
  qcow2/overlaps: Protect snapshot table
  qcow2/overlaps: Protect inactive L1 tables
  qcow2/overlaps: Protect inactive L2 tables
  qcow2: Use new metadata overlap check function

 block/Makefile.objs    |   3 +-
 block/qcow2-cluster.c  |  13 ++
 block/qcow2-overlap.c  | 396 +++++++++++++++++++++++++++++++++++++++++++++++++
 block/qcow2-refcount.c | 202 ++++++++++---------------
 block/qcow2-snapshot.c | 105 ++++++++++++-
 block/qcow2.c          | 131 ++++++++++------
 block/qcow2.h          |  13 ++
 7 files changed, 691 insertions(+), 172 deletions(-)
 create mode 100644 block/qcow2-overlap.c

-- 
2.1.0