Re: [Qemu-block] [Qemu-devel] Persistent bitmaps for non-qcow2 formats

[John Snow]

On 08/25/2017 09:44 AM, Max Reitz wrote:
> On 2017-08-25 02:55, John Snow wrote:
>> Sorry in advance for :words: ...
>>
>> On 08/23/2017 02:04 PM, Vladimir Sementsov-Ogievskiy wrote:
>>> 23.08.2017 11:59, Vladimir Sementsov-Ogievskiy wrote:
>>>> 22.08.2017 22:07, John Snow wrote:

[snip]

>>>>
>>>> Should there be some problems with internal snapshots and other things?
> 
> I'd suspect you get exactly the same problems when using internal
> snapshots together with backing files.  Imagine a newly created overlay
> file and taking a snapshot.  This should give you exactly the same
> issue, doesn't it?
> 
>>>>
>>>>
>>>>
>>> Hm. looks like that this backing file should not only receive all reads
>>> and writes, but almost everything ->bdrv_ handlers, except bitmap
>>> related of course.
> 
> How so?  Shouldn't it just work like a backing file, except it also
> receives writes instead of just reads?
> 
>>>                    This doesn't seems simple to implement. Especially if
>>> imaging some not-raw feature-full format under this thin qcow2 layer..
>>> Or we can restrict this RW backing file to be raw-only?
>>>
>>>
>>
>> The idea would really be to support any arbitrary data store, so I
>> wouldn't want to restrict it to just raw.
>>
>> You're right though, this might be a kind of messy approach.
>>
>> [From your other mail:]
>>
>>>
>>> So, anyway, I see only two differences (from the outside) between this 
>>> approach and just a separate bitmap-only qcow2 without a data:
>>>
>>
>> It's very delicately similar, yes.
>>
>>> 1. in RW-backing approach qcow2-bitmap file has a link to data file (as a 
>>> backing). It looks good.
> 
> And this is rather important to me.
> 

Good to know. Some good solid opinions to work around. ;)

>> Right. The information necessary to establish a link between the bitmap
>> data and the data being described is fully contained within a file fully
>> specified by the QCOW2 spec.
>>
>>> 2. in RW-backing approach qcow2-bitmap file is a top of the virtual disk, 
>>> in separate-file approach it is an option of the real data drive. In my 
>>> opinion the second is more clean for users ("to add this feature you should 
>>> use other file as your disk" vs "to add this feature you should add an 
>>> option to your disk description")
> 
> I'd argue it's rather: "You cannot use this feature unless your format
> supports it.  The only format supporting persistent bitmaps currently is
> qcow2.  To use persistent bitmaps with other formats you can attach them
> as R/W backing files to an empty qcow2 file, though."
> 
> So the difference is that you are saying it's a feature that is added to
> a non-qcow2 image whereas I'm saying it's a feature that only a qcow2
> image can provide (currently).
> 
>> This puts us a little closer to the original idea that was rejected by
>> Kevin at the time. To recap:
>>
>> "1": Use qcow2 as a container. This was rejected because we didn't want
>> qcow2 containing data with no semantic relationship to the qcow2
>> container or to each other. The way it sounds like you're proposing it,
>> though, it would be one-qcow2-with-bitmaps-per-drive, so the data would
>> at least stay strictly related, but it would be meaningless outside of
>> QEMU itself. I think this is something that Kevin wanted to avoid, but I
>> can't speak for him.
>>
>> It's certainly not beyond the realm of management software to remember
>> to correlate a qcow2 metadata file alongside its actual data stores
>> whenever it needed to do so, but it does mean the introduction of a
>> feature that essentially requires the use of management software, which
>> sees resistance in the community at times.
>>
>> In this model, you'd probably have the raw drive at the top, with the
>> qcow2-with-bitmaps as a child node with some kind of new named child
>> relationship. All IO stays at the root node, but the bitmap method
>> handlers would know to look for this special bitmap-child. It shouldn't
>> be too hard to implement.
> 
> I'd still like to throw in how much I dislike this approach, and I can't
> really think of a way to make it palatable to me.  Not even "just write
> the file name of the image the bitmaps cover into the qcow2 file" sounds
> good to me, because then it still is basically unrelated data.
> 

Understood. It's something I'd like to avoid too, but I have some real
concerns about implementation of that semantic link.

> The only approach that I might see myself liking is to indeed add a flag
> or whatever to say a qcow2 backing image is supposed to be R/W; and then
> (after somehow verifying that the qcow2 image itself is empty) just make
> qemu interpret this as "load the backing file as the real disk and
> attach the qcow2 image as a 'metadata' child" or whatever.  But I fear
> this gets uglier and uglier because how qemu loads the files will then
> depend on whether the overlay is empty or not, and this may be very
> confusing.
> 

Right, it makes opening a little more convoluted than it normally is.

"Oh look, it's empty and it has a RW backing file. Please stand by as we
open the "RW backing file" and install it as our parent!"

It may actually not be so bad, but it does add a complexity...

>>> I think (may be I'm not right) loading bitmaps from additional 
>>> qcow2-only-bitmaps file is simpler to implement (it will be specified in 
>>> command line in drive options, like bitmaps_file=/path/to/it and then 
>>> attached directly to BlockDriverState). The only drawback of simple 
>>> qcow2-bitmap file  is that it has not a link inside it to the data file 
>>> (like backing). We can ignore it, or we can implement this link as a 
>>> separate extension to qcow2.
>>
>> Yes, definitely easier to implement as you say.
>>
>> The hard part is going to come in defining that semantic link. At this
>> point, the only difference between the approaches is whether or not we
>> allow the qcow2 to point to the implementation of the data;
>>
>> (1) The qcow2 is referenced by name from the CLI as an option to the
>> other drive.
>>
>> (2) The qcow2 is referenced by name on the CLI, and its backing file
>> field intuits the location of the implementation storage.
>>
>>
>> In (1), we avoid saving or specifying the relationship between these two
>> data stores in any way. This is certainly easy to do, and will save us
>> some headache on the CLI. As a downside, we now have random orphaned
>> files that aren't very interesting or useful on their own. The
>> likelihood for desync between metadata and data increases. The use of
>> management software is all but necessitated.
>>
>> In (2) We have to now specify, with a dizzying long list of
>> possibilities, the location of the implementation data. qcow2 only has a
>> filename for backing files presently, but this is likely inadequate.
>> What if the data store isn't a locally kept file? What if it's a socket,
>> or a stream, or literally anything else?
> 
> I don't see the difference.  In (1), your data image gets a "bitmap" or
> "metadata" child.  In (2), your qcow2 image gets the usual "backing"
> child, or maybe call it "passthrough" or whatever, if you want to make
> the difference more explicit than just passing an option to the qcow2
> image to pass writes to its backing file.
> 

The difference in the relationship in-memory is actually kind of
uninteresting.

The difference as I see it is primarily how we specify the relationship
between the qcow2 and the implementation storage; in (1) It's defined
on-disk, and in (2) It's defined via CLI only, so

(1) Incurs a cost of having to define the link syntax (possibly causing
a rather qemu-specific syntax), and
(2) Avoids that cost, but leaves the data on-disk unrelated, which you hate.

>> We'd have to develop a new syntax for specifying these resources that
>> can be stored in a qcow2 file,
> 
> It's called the json-pseudo-protocol and was developed exactly for this.
> 

That's what I was hinting at for "or otherwise co-opt an existing
syntax" but I was unaware that it was intended for "exactly" this.

Do we actually use it in any on-disk format, currently? qcow2 only lets
you specify simple filenames in the qcow2 metadata, right?

>>                                or otherwise co-opt an existing syntax
>> in-use by QEMU. This syntax would likely be useful only to QEMU, which
>> would steer the qcow2 format in a direction not too useful by other
>> emulators, and qcow2 is an open format, so we may want to avoid this.
> 
> Storing a file name in the backing link field that cannot be interpreted
> by other programs is in my opinion still very much better than not
> storing any information whatsoever, because in the former case other
> programs can at least say "sorry, I have no idea what this means" (or
> maybe they can indeed interpret it, who knows), whereas in the latter
> they may not even know that the qcow2 image is incomplete.
> 

I don't disagree personally, but I seem to recall that Kevin was adamant
that the qcow2 bitmap extension should remain useful and semantically
meaningful to third parties, so I try to keep that in mind. Maybe I
should let him chime in instead of try to "concern troll" my own
suggestions into the ground.

> Also note that we are making an effort to be able to generate plain file
> names (such as URLs which should be usable by other programs) whenever
> possible.
> 

Noted. Do we have a useful discriminator anywhere that allows us to
easily check if a filename/locator/URI/whatever is in an accepted
format, or if we still have QEMU-specific garbage?

We could always just disallow QEMU-specific protocol-talk from getting
written and allow this only for configurations that QEMU understands to
be universal...

>> I feel like what will make the difference between heading down either
>> path would be helped along by answers to these questions:
>>
>> - What type of data stores do we wish to support with bitmaps? Simple
>> file-based ones, or the full spectrum of all types? Only qcow2, and to
>> hell with people who ask for otherwise?
> 
> I don't really know how this question relates to the issue other than
> "If we only want to support qcow2, the whole discussion is moot; since
> this discussion exists, we apparently do want to support something other
> than qcow2 at some point."
> 

Yeah, I mean, if the answer is a strong no here we can avoid the rest of
the discussion. Worthwhile to know, right?

> Well, OK.  The main argument against supporting anything but qcow2 is
> "if you want features, use qcow2; and we are working on making qcow2 as
> fast as possible."  I think that's a very good argument still.  At some
> point I (and probably others, too) had the idea of making qcow2 files in
> raw layout: Have the data as a blob, just like a raw file, padded by
> metadata around it.  An autoclear flag would specify that the qcow2 file
> is in this format, and if so, you could simply access it like a raw file
> and should have exactly the same speed as a raw file.  Maybe that would
> solve this whole issue, too?
> 
> And as for non-file based backing files, see above.
> 
>> - How important is it that the qcow2 remains a fully independent file
>> capable of describing its own relationship to the data?
> 
> Technically not important.  To me, very important.
> 
>> - Is it OK to allow robust, QEMU-specific data descriptors in a qcow2 file?
> 
> Depends.  At least Red Hat's QA does use json file names, so there's
> already that.
> 
> I think QEMU-specific is OK as long as it still makes sense, as long as
> there is no other way, and as long as there is some kind of
> documentation still.
> 
> Technically, you can always put QEMU-specific stuff into the qcow2
> specification and thus make it generic.
> 
>> Where I sit:
> 
> [...]
> 
>> - I don't think Kevin will like the idea of us using qcow2 as a
>> container that does not get treated as a first class citizen in the
>> backing chain, but it's the easiest option to implement and avoids a lot
>> of the syntax-of-backing-storage questions.
> 
> I'd like to throw in that I was not at all opposed to Fam's approach of
> having an independent format (tar, wasn't it?) just for storing bitmaps.
>  I know Kevin was, but I don't quite remember why.  Probably because it
> was a real format, but a very strange one, and then implementing it
> through the block layer was weird...?
> 
> OTOH, there is one issue I have with the R/W backing approach: Every
> request to the raw file has to go through the qcow2 layer.  And since
> you probably want to use raw because of the speed, this is not so nice.
> 

We probably could fix that by changing the relationship so that it isn't
*really* a backing file, but that maybe creates other problems.

> Max
>