Re: [Qemu-devel] Incremental backup call

[John Snow]

Thank you, Kashyap!

For a general audience, let me annotate this just a little bit. I'll try
to avoid being too partial in this email, and will follow up with some
opinions in a separate email, if any.

If I have misrepresented the spirit of a concern or objection, please do
correct my annotations.

On 02/23/2016 09:25 AM, Kashyap Chamarthy wrote:
> On Tue, Feb 16, 2016 at 02:02:58PM +0000, Stefan Hajnoczi wrote:
>> Hi,
>> There are several ongoing efforts to implement incremental
>> backup-related features.
>>
>> Let's have a voice/video conference to get everyone on the same page,
>> avoid duplicated work, and get patches merged faster.
>>
>> Agenda:
>>
>>  * External incremental backup API.  Summarize requirements common to
>> third-party backup appliances and agree on suitable API direction.
>>
>>  * Persistent dirty bitmaps.  Agree on how qcow2 and/or qbm will hold
>> bitmaps in various use cases.
>>
>>  * Anything else?
> 
> I was in a listen-only mode on the call, here's some quick minutes
> (read: haphazard scribbling) so please excuse grammar
> thinkos/typos/missed points.  Feel free to correct/adjust.
> 
> -----------------------------------------------------------------------
> - [denis-lunev] Migration support for dirty bitmaps?
> - [stefanha] One discussion that might need to reset things is Fam's
>   QEMU QBM (Qemu Bit Map)

QBM is a proposed image format that is essentially an on-disk wrapper
around existing formats. It's a plaintext JSON file that describes a
disk image resource and an existing format (e.g. raw, qcow2, etc) and a
number of raw bitmap data files.

It is essentially a "filter" format, but one that exists on-disk.

> - [stefanha] Incremental backups _should_ be possible with RAW
> - [jsnow] Apart from just from having incremental backups for RAW
>   devices, any reason why qcow2 _cannot_ be the bitmap provider?

One of the existing arguments against augmenting non-native image
formats (e.g. adding features to RAW) is complexity of maintenance and
wanting to bolster the strength of the qcow2 format.

The refrain is "If qcow2 can't do it, we should fix qcow2."

However, I wonder if there are any cases we wish to support incremental
backup for that do not or cannot use .qcow2 as the root format for the
drive -- principally SAN devices and other networked block devices where
qcow2 is not the primary handle to the resource tree.

> - [kwolf] What are the actual requirements?
> - [stefanha] Vladimir's series that modified qcow2 - the idea was that
>   you don't _have_ to use qcow2 format, but you could use it as a bitmap
>   container for other images.

The original proposal for bitmap persistence involved using .qcow2 as a
container format to store bitmaps for *any* image, not just ones related
to this particular qcow2.

The one that was merged recently pares down this idea into storing only
bitmaps related to that qcow2.

> - [kwolf] Something about VM state internal snapshots: where the VM
>   state is saved into qcow2 file:
>    - Don't store bitmaps in a qcow2 file

Kevin feels that storing VM snapshots inside of .qcow2 files was a
mistake, because it contains data not related necessarily to the image
stored inside of the qcow2. His thought was that storing any bitmaps
inside of a qcow2 is similar to the mistake that was made with VM snapshots.

There is no major opposition to storing bitmaps that describe a qcow2
inside of that qcow2.

>    - Stack the qcow2 on the specific image that the bitmap is for

Instead of using QBM as our filter format, perhaps qcow2 can be extended
to fulfill the same role, which avoids us cooking up an entirely new
primary/first-party image format that we'd have to support.

>    - [stefanha] What happens when there's guest I/O (?) 

qcow2 does not currently support a mode where the top-layer is a filter.
writes currently always land at the top layer, which would be
undesirable for a "bitmap filter."

> - [kwolf] The question is: if it wouldn't be simple to extend qcow2 - to
>   forward all writes to the backing file
> - If we _don't_ want to use qcow2 to store this:
> - [kwolf] Concern is consistency: doing one thing for qcow2; and other
>   thing for other formats?

We are concerned over the redundancy of having both per-image-format
bitmap saving routines AND having a global bitmap saving mechanic.

To date, we have approved a qcow2 extension for bitmap saving, and Denis
Lunev at al either already have or are prepared to write a similar
extension for the parallels image format.

Is it too messy or complicated to support qcow2 native saving, parallels
saving, AND an "any-format" saving mechanism? Will this lead to too much
code duplication and an undue maintenance burden caused by a lack of
planning?

> - [eblake] This is not the first time we're adding something on top of
>   raw images
>    - [eblake] bitmap on top of RAW - this concept has been floating on
>      the list for several years
>       - https://lists.nongnu.org/archive/html/qemu-devel/2013-08/msg03682.html
>       - [kwolf] What is the use case?
>          - Backing files with raw images
>          - If you have a raw image file, you can use backing files with
>            it; if at some point in time

Desires to augment RAW in QEMU have existed for a long time, but qcow2's
performance has continued to improve, lessening some of the technical
merit to past justifications. Are there any hard technical reasons why
RAW *must* still be augmented?

We don't have a smoking gun.

> - [eblake] Mentioned LUKS encryption of danpb (in comparision with QBM
>   work?)
>    - A seperate driver (general purpose) 'luks' format driver which can
>      be layered over any other existing block driver.
>    - Embedded LUKS inside qcow2
>       - it's encrypted as _part_ of the qcow2 file
>    - A general purpose 'luks' format driver which can be layered over
>      any other existing block driver.
>    - To differentiate intentionally encrypted as part of qcow2 vs. the
>      guest data
>    - [kwolf] One important difference: Dan didn't introduce a new file
>      _format_ for qcow2.
>   - [eblake] In a sense, LUKS _is_ a new file format - but indeed it's
>     interoperating with existing format
> 
> - [stefanha] Is the least controversial part: getting the Qcow2 support
>   in that Vladimir is working on?
> - [kwolf] Concern: is the relationship with QBM -- I don't want to end
>   up with duplicated things at the end
>    - Once we accept this API, we can't change it

Can we progress on the qcow2-native bitmap persistence feature for now
and worry about QBM later? There are concerns that by solidifying the
qcow2 method now, we may forever prohibit ourselves from a clean,
non-redundant universal persistence solution.

> - [denis-lunev] A lot of resources/time has been invested in this.
>   Specification about bitmaps.  Version-17 for this spec is too much.
> - [kwolf] If you go forward with qcow2 path, then, we're not going to do
>   the QBM approach.
> - [fam] Qcow2 extension has made its progress
>    - We should go ahead and make the extension in qcow2 and see how it
>      goes
>    - People will start complaining about missing features in raw (?)
>    - Concern: in certain protocols like Ceph, people would tend to use
>      Raw images, without any tools on top
> - [stefanha] If we do what Kevin told: qcow2 will have a node - where
>   the writes will be forwarded to backing file, what's holding up there
>   (?)
>    - You have a qcow2 file that doesn't have much data, except metadata
>      and including bitmap
> - [jsnow] We should move forward with qcow2 persistence approach.
>    - We can re-discuss the merits of duplication again

Consensus appears to be to move forward with per-format bitmap saving
for now (starting with qcow2, parallels perhaps to follow.)

We can re-investigate the QBM/qcow2 as a filter-format argument later
after more client use cases come to light. We can debate about the
merits/demerits of code duplication between a per-format persistence and
a global persistence when the need arises.

The qcow2 format is an obvious use case that is obviously desired and
one that we have the most justification for, so it is probably
unreasonable to delay work pending future debates.

> - [stefanha] Question for Eric: In terms of committing an API, client
>   applications, external plugins for backup/ related libvirt work?
>    - [eblake] From libvirt's perspective, the biggest thing we need for
>      2.6 is having QMP 'blockdev-add' for everything:
>       - We want to get Ceph, NBD and Gluster all of those under QAPI, to
>         be able to programatically address them from libvirt
>          - We currently sometimes fallback to HMP
>       - [kwolf] I don't think it's possible for 2.6 yet.  So far, we
>         have NBD support on the list
>       - [eblake] Going to look at 'blockdev-add' to see what's missing
>         and what needs to be added
> - [eblake] Ability to track persistent node names in libvirt XML
>     - We're at the point now that every node creation can have a name.
>     - libvirt does have some work to do take advantage of node names,
>       remembering what names it has assigned
> -----------------------------------------------------------------------
> 
> 

--js