Re: query dirty areas according to bitmap via QMP or qemu-nbd

[Eric Blake]

On Mon, Jul 29, 2024 at 03:51:17PM GMT, Fiona Ebner wrote:
> > In particular, tests/qemu-img-bitmaps gives the magic decoder ring:
> > 
> > | # x-dirty-bitmap is a hack for reading bitmaps; it abuses block status to
> > | # report "data":false for portions of the bitmap which are set
> > | IMG="driver=nbd,server.type=unix,server.path=$nbd_unix_socket"
> > | nbd_server_start_unix_socket -r -f qcow2 \
> > |     -B b0 -B b1 -B b2 -B b3 "$TEST_IMG"
> > | $QEMU_IMG map --output=json --image-opts \
> > |     "$IMG,x-dirty-bitmap=qemu:dirty-bitmap:b0" | _filter_qemu_img_map
> > 
> > meaning the map output includes "data":false for the dirty portions
> > and "data":true for the unchanged portions recorded in bitmap b0 as
> > read from the JSON map output.
> > 
> 
> Oh, I didn't think about checking the NBD block driver for such an
> option :) And thank you for all the explanations!

Glad it helped.  The 'x-' in the name is a giveaway that the command
is not intended for public use, but for development, it makes iotests
easier to write, so I see no reason to drop it.

> 
> >>
> >> If it is not currently possible, would upstream be interested too in the
> >> feature, either for QMP or qemu-nbd?
> > 
> > Improving qemu-img to get at the information without quite the hacky
> > post-processing deciphering would indeed be a useful patch, but it has
> > never risen to the level of enough of an itch for me to write it
> > myself (especially since 'nbdinfo --map's output works just as well).
> > 
> 
> I might just go with the above for now, but who knows if I'll get around
> to this some day. Three approaches that come to mind are:
> 
> 1. qemu-img bitmap --dump
> 
> Other bitmap actions won't be supported in combination with NBD.

This seems like an independently useful option (for any image with
embedded bitmaps, get at the contents of that bitmap).  I'm not sure
how much code you could reuse from the 'qemu-img map', or if it is
better to just write the iteration loops from scratch.  More
interesting is the observation that since NBD metacontexts are NOT
visible as a local bitmap (but rather as block status information), it
may still not do what you want for NBD connections.

> 
> 2. qemu-img map --bitmap NAME
> 
> Should it use a dedicated output format compared to the usual "map"
> output (both human and json) with just "start/offset + length + dirty
> bit" triples?

Again, this would work well with local qcow2 images, but I'm not sure
if it would translate nicely into NBD client images without more code
in block/nbd.c to expose a metacontext as a bitmap rather than its
current use of feeding block status.

> 
> 3. qemu-nbd --map CONTEXT
> 
> With only supporting one context at a time? Would be limited to NBD of
> course which the other two won't be.

The other two would be hard to implement for NBD, while this one is
hard to implement for qcow2.  It might be less hacky than
x-dirty-bitmap, but also starts duplicating efforts with what nbdinfo
can already do.

> 
> 
> All would require connecting to the NBD export with the correct meta
> context, which currently means using x_dirty_bitmap internally. So would
> that even be okay as part of a non-experimental command, or would it
> require to teach the NBD client code to deal with multiple meta contexts
> first?

If you want qemu-img to treat both qcow2 bitmaps and NBD metacontexts
transparently from the same command line, we'll definitely need some
work on how to make block/nbd.c expose an NBD metacontext as a bitmap
(separately from how the existing x-dirty-bitmap just repurposes what
is plugged into the block_status callbacks).  But handling only one
metacontext at a time, rather than having to do multiple in parallel,
is probably okay.  When you start reading more than one context at
once, you have to worry about things like the two contexts returning
different lengths of information ("base:allocation" might tell you
about a 64k hole while qemu:dirty-bitmap:XXX tells you about a 1M
dirty region - but consolidating that into contiguous extents of
combined output information is tricky, especially if you don't want to
re-query status you already know about one context but not the other).

> 
> Best Regards,
> Fiona
> 

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.
Virtualization:  qemu.org | libguestfs.org