[Qemu-block] [PATCH v2.5 00/10] block: incremental backup transactions

[John Snow]

I am not prepared to send a v3 on this, primarily because I am still 
waffling on whether or not to do the code motion patch that is present 
in patch #8 of v2 of this series.

However, for the purposes of testing, reviewers may find it convenient 
to have a new version of this series that applies cleanly in concert 
with v6 of the transactionless series, so I am presenting an informal 
"version 2.5" of this series.

Patches are available on github: 
https://github.com/jnsnow/qemu/commits/incremental-transactions

And here is the v2.5 patchset diff:

Key:
[----] : patches are identical
[####] : number of functional differences between upstream/downstream patch
[down] : patch is downstream-only
The flags [FC] indicate (F)unctional and (C)ontextual differences, 
respectively

001/10:[0002] [FC] 'qapi: Add transaction support to block-dirty-bitmap 
operations'
002/10:[0021] [FC] 'iotests: add transactional incremental backup test'
003/10:[----] [--] 'block: rename BlkTransactionState and BdrvActionOps'
004/10:[----] [--] 'block: re-add BlkTransactionState'
005/10:[0010] [FC] 'block: add transactional callbacks feature'
006/10:[----] [--] 'block: add refcount to Job object'
007/10:[----] [-C] 'block: add delayed bitmap successor cleanup'
008/10:[0051] [FC] 'qmp: Add an implementation wrapper for qmp_drive_backup'
009/10:[0001] [FC] 'block: drive_backup transaction callback support'
010/10:[0033] [FC] 'iotests: 124 - transactional failure test'

01: Removed "Not yet implemented" line from bitmaps.md
02: Rebase fallout
05: Removed superfluous deletion loop pointed out by Max
08: Fallout from removing the code motion patch,
    new forward declaration.
09: Fallout from removing the code motion patch,
    new forward declaration.
10: Fallout from the rebase.


There are some more changes that need to happen to #10, primarily a bit 
more refactoring after the changes made to how images are checked.

On 03/27/2015 03:19 PM, John Snow wrote:
> requires: address@hidden
>            "[PATCH v4 00/20] block: transactionless incremental backup series"
>
> This series adds support for incremental backup primitives in QMP
> transactions. It requires my transactionless incremental backup series,
> currently at v4.
>
> Patch 1 adds basic support for add and clear transactions.
> Patch 2 tests this basic support.
> Patches 3-4 refactor transactions a little bit, to add clarity.
> Patch 5 adds the framework for error scenarios where only
>      some jobs that were launched by a transaction complete successfully,
>      and we need to perform context-sensitive cleanup after the transaction
>      itself has already completed.
> Patches 6-7 add necessary bookkeeping information to backup job
>      data structures to take advantage of this new feature.
> Patch 8 just moves code.
> Patch 9 modifies qmp_drive_backup to support the new feature.
> Patch 10 implements the new feature for drive_backup transaction actions.
> Patch 11 tests the new feature.
>
> Lingering questions:
>   - Is it worth it to add a post-transaction completion event to QMP that
>     signifies all jobs launched by the transaction have completed? This
>     would be of primary interest to libvirt, in particular, but only as
>     a convenience feature.
>
> Thank you,
> --John Snow
>
> v2:
>
> Key:
> [----] : patches are identical
> [####] : number of functional differences between upstream/downstream patch
> [down] : patch is downstream-only
> The flags [FC] indicate (F)unctional and (C)ontextual differences, respectively
>
> 001/11:[0009] [FC] 'qapi: Add transaction support to block-dirty-bitmap 
> operations'
> 002/11:[0036] [FC] 'iotests: add transactional incremental backup test'
> 003/11:[down] 'blockdev: rename BlkTransactionState and BdrvActionOps'
> 004/11:[down] 'block: re-add BlkTransactionState'
> 005/11:[0274] [FC] 'block: add transactional callbacks feature'
> 006/11:[0004] [FC] 'block: add refcount to Job object'
> 007/11:[0048] [FC] 'block: add delayed bitmap successor cleanup'
> 008/11:[down] 'block: move transactions beneath qmp interfaces'
> 009/11:[0084] [FC] 'qmp: Add an implementation wrapper for qmp_drive_backup'
> 010/11:[0050] [FC] 'block: drive_backup transaction callback support'
> 011/11:[0004] [FC] 'iotests: 124 - transactional failure test'
>
>   01: Fixed indentation.
>       Fixed QMP commands to behave with new bitmap_lookup from
>         transactionless-v4.
>       2.3 --> 2.4.
>   02: Folded in improvements to qemu-iotest 124 from transactional-v1.
>   03: NEW
>   04: NEW
>   05: A lot:
>       Don't delete the comma in the transaction actions config
>       use g_new0 instead of g_malloc0
>       Phrasing: "retcode" --> "Return code"
>       Use GCC attributes to mark functions as unused until future patches.
>       Added some data structure documentation.
>       Many structure and function renames, hopefully to improve readability.
>       Use just one list for all Actions instead of two separate lists.
>       Remove ActionState from the list upon deletion/decref
>       And many other small tweaks.
>   06: Comment phrasing.
>   07: Removed errp parameter from all functions introduced by this commit.
>       bdrv_dirty_bitmap_decref --> bdrv_frozen_bitmap_decref
>   08: NEW
>   09: _drive_backup --> do_drive_backup()
>       Forward declarations removed.
>   10: Rebased on top of drastically modified #05.
>       Phrasing: "BackupBlockJob" instead of "BackupJob" in comments.
>   11: Removed extra parameters to wait_incremental() in
>         test_transaction_failure()
>
> John Snow (11):
>    qapi: Add transaction support to block-dirty-bitmap operations
>    iotests: add transactional incremental backup test
>    block: rename BlkTransactionState and BdrvActionOps
>    block: re-add BlkTransactionState
>    block: add transactional callbacks feature
>    block: add refcount to Job object
>    block: add delayed bitmap successor cleanup
>    block: move transactions beneath qmp interfaces
>    qmp: Add an implementation wrapper for qmp_drive_backup
>    block: drive_backup transaction callback support
>    iotests: 124 - transactional failure test
>
>   block.c                    |   65 +-
>   block/backup.c             |   29 +-
>   blockdev.c                 | 1599 ++++++++++++++++++++++++++++----------------
>   blockjob.c                 |   18 +-
>   include/block/block.h      |   10 +-
>   include/block/block_int.h  |    8 +
>   include/block/blockjob.h   |   21 +
>   qapi-schema.json           |    6 +-
>   tests/qemu-iotests/124     |  170 +++++
>   tests/qemu-iotests/124.out |    4 +-
>   10 files changed, 1313 insertions(+), 617 deletions(-)