Re: query dirty areas according to bitmap via QMP or qemu-nbd

[Fiona Ebner]

Am 26.07.24 um 17:38 schrieb Eric Blake:
> On Fri, Jul 26, 2024 at 04:16:41PM GMT, Fiona Ebner wrote:
>> Hi,
>>
>> sorry if I'm missing the obvious, but is there a way to get the dirty
>> areas according to a dirty bitmap via QMP? I mean as something like
>> offset + size + dirty-flag triples. In my case, the bitmap is also
>> exported via NBD, so same question for qemu-nbd being the client.
> 
> Over QMP, no - that can produce a potentially large response and
> possible long time in computing the data, so we have never felt the
> need to introduce a new QMP command for that purpose.  So over NBD is
> the preferred solution.
> 
>>
>> I can get the info with "nbdinfo --map", but would like to avoid
>> requiring a tool outside QEMU.
> 
> By default, QEMU as an NBD client only reads the "base:allocation" NBD
> metacontext, and is not wired to read more than one NBD metacontext at
> once (weaker than nbdinfo's capabilities).  But I have intentionally
> left in a hack (accessible through QMP as well as from the command
> line) for connecting a qemu NBD client to an alternative NBD
> metacontext that feeds the block status, at which point 2 bits of
> information from the alternative context are observable through the
> result of block status calls.  Note that using such an NBD connection
> for anything OTHER than block status calls is inadvisable (qemu might
> incorrectly optimize reads based on its misinterpretation of those
> block status bits); but as long as you limit the client to block
> status calls, it's a great way to read out a "qemu:dirty-bitmap:..."
> metacontext using only a qemu NBD client connection.
> 
> git grep -l x-dirty-bitmap tests/qemu-iotests
> 
> shows several of the iotests using the backdoor in just that manner.
> In particular, tests/qemu-img-bitmaps gives the magic decoder ring:
> 
> | # x-dirty-bitmap is a hack for reading bitmaps; it abuses block status to
> | # report "data":false for portions of the bitmap which are set
> | IMG="driver=nbd,server.type=unix,server.path=$nbd_unix_socket"
> | nbd_server_start_unix_socket -r -f qcow2 \
> |     -B b0 -B b1 -B b2 -B b3 "$TEST_IMG"
> | $QEMU_IMG map --output=json --image-opts \
> |     "$IMG,x-dirty-bitmap=qemu:dirty-bitmap:b0" | _filter_qemu_img_map
> 
> meaning the map output includes "data":false for the dirty portions
> and "data":true for the unchanged portions recorded in bitmap b0 as
> read from the JSON map output.
> 

Oh, I didn't think about checking the NBD block driver for such an
option :) And thank you for all the explanations!

>>
>> If it is not currently possible, would upstream be interested too in the
>> feature, either for QMP or qemu-nbd?
> 
> Improving qemu-img to get at the information without quite the hacky
> post-processing deciphering would indeed be a useful patch, but it has
> never risen to the level of enough of an itch for me to write it
> myself (especially since 'nbdinfo --map's output works just as well).
> 

I might just go with the above for now, but who knows if I'll get around
to this some day. Three approaches that come to mind are:

1. qemu-img bitmap --dump

Other bitmap actions won't be supported in combination with NBD.

2. qemu-img map --bitmap NAME

Should it use a dedicated output format compared to the usual "map"
output (both human and json) with just "start/offset + length + dirty
bit" triples?

3. qemu-nbd --map CONTEXT

With only supporting one context at a time? Would be limited to NBD of
course which the other two won't be.


All would require connecting to the NBD export with the correct meta
context, which currently means using x_dirty_bitmap internally. So would
that even be okay as part of a non-experimental command, or would it
require to teach the NBD client code to deal with multiple meta contexts
first?

Best Regards,
Fiona