Re: [Qemu-devel] [RFC PATCH 00/16] Qemu Bit Map (QBM) - an overlay format for persistent dirty bitmap

[Fam Zheng]

(I'm Cc'ing a few more people here just in case they have different visions
about raw image use cases.)

On Mon, 02/22 15:24, Kevin Wolf wrote:
> Am 26.01.2016 um 11:38 hat Fam Zheng geschrieben:
> > This series introduces a simple format to enable support of persistence of
> > block dirty bitmaps. Block dirty bitmap is the tool to achieve incremental
> > backup, and persistence of block dirty bitmap makes incrememtal backup 
> > possible
> > across VM shutdowns, where existing in-memory dirty bitmaps cannot survive.
> > 
> > When user creates a "persisted" dirty bitmap, the QBM driver will create a
> > binary file and synchronize it with the existing in-memory block dirty 
> > bitmap
> > (BdrvDirtyBitmap). When the VM is powered down, the binary file has all the
> > bits saved on disk, which will be loaded and used to initialize the 
> > in-memory
> > block dirty bitmap next time the guest is started.
> > 
> > The idea of the format is to reuse as much existing infrastructure as 
> > possible
> > and avoid introducing complex data structures - it works with any image 
> > format,
> > by gluing it together plain bitmap files with a json descriptor file. The
> > advantage of this approach over extending existing formats, such as qcow2, 
> > is
> > that the new feature is implemented by an orthogonal driver, in a format
> > agnostic way. This way, even raw images can have their persistent dirty
> > bitmaps.  (And you will notice in this series, with a little forging to the
> > spec, raw images can also have backing files through a QBM overlay!)
> > 
> > Rather than superseding it, this intends to be coexistent in parallel with 
> > the
> > qcow2 bitmap extension that Vladimir is working on.  The block driver 
> > interface
> > changes in this series also try to be generic and compatible for both 
> > drivers.
> 
> So as I already told Fam last week, before we discuss any technical
> details here, we first need to discuss whether this is even the right
> thing to do. Currently I'm doubtful, as this is another attempt to
> introduce a new native image format in qemu.
> 
> Let's recap the image formats and what we tell users about them today:
> 
> * qcow2: This is the default choice for disk images. It gives you access
>   to all of the features in qemu at a good performance. If it doesn't
>   perform well in your case, we'll fix it.
> 
> * raw: Use this when you need absolute performance and don't need any
>   features from an image format, so you want to get any complexity just
>   out of the way and pass requests as directly as possible from the
>   guest device to the host kernel.
> 
> * Anything else: Only use them to convert into raw or qcow2.
> 
> Now using bitmaps is clearly on the "features" side, which suggests that
> qcow2 is the format of choice for this. If you want to introduce a new
> format, you need to justify it with evidence that...
> 
> 1. there is a relevant use case that qcow2 doesn't cover
> 2. qcow2 can't be fixed/enhanced to cover the use case
> 
> The one thing that people have claimed in the past that qcow2 can't
> provide is enough performance. This is where QED tried to come in and
> promised a compromise between performance (then a bit faster than qcow2)
> and features (almost none, but supports backing files). We all know that
> it was a failure because you had to sacrifice features and still the
> idea that qcow2 couldn't be fixed was wrong, so today we have a QED
> driver that is much slower than qcow2 despite having less features.
> 
> Now for QBM. First, let's have a look at the image format that it can be
> used with. qcow2 doesn't need it if we continue with Vladimir's
> extension. Other non-raw formats are only supposed to be used for
> conversion. The only thing that's really left is raw.

Yes, I agree with this point.

> Now adding a
> feature only for raw, as a compromise between features and performance,
> looks an awful lot like what QED tried. We don't want to go there.
> 
> Even if we wanted to support persistent dirty bitmaps with raw images
> (which has to be discussed based on use cases), it's still questionable
> whether we need a new image format with JSON descriptor files instead of
> just raw bitmaps that can be added with a QMP command.
> 

I don't think QMP interface alone is enough, in persistent backup use case,
when starting a guest, command line interface is more appropriate to continue
dirty trackings that were enabled during shutdown.

I'd justify in two parts, one is "why" and the other is "how".

So to answer why.  The reason I worked on QBM is because I feel it wrong to
leaving raw behind. Ceph and LVM users use raw format. You could technically
use qcow2 with ceph but that is discouraged[1] or even refused by openstack[2].
We've seen qcow2 on top of LVs but that is not the dominance.

The scope of "features" for which we tell users they have to use qcow2 should
those that are format specific, not "block features" in general.  Backing file,
internal/external snapshot, thin provisioning, compression and encryption are
all great examples of format features, whereas things including throttling,
statistics, migration, mirroring and backing up are IMHO not.  Actually we
already support snapshotting a raw image, with an qcow2 overlay.  We've even
implemented non-persistent incremental backup for raw today, through
drive-backup.  If we will decide qcow2 is the only possible format that can do
persistent backup, I'm not really a huge fan of it.

Then "how"?

Actually, I thought we could do it in a way similar to quorum. The way quorum
driver works is by specifying tediously long options. A snippet from
qemu-iotests to build a quorum driver with 3 children is like this:

    quorum="driver=raw,file.driver=quorum,file.vote-threshold=2"
    quorum="$quorum,file.children.0.file.filename=$TEST_DIR/1.raw"
    quorum="$quorum,file.children.1.file.filename=$TEST_DIR/2.raw"
    quorum="$quorum,file.children.2.file.filename=$TEST_DIR/3.raw"
    quorum="$quorum,file.children.0.driver=raw"
    quorum="$quorum,file.children.1.driver=raw"
    quorum="$quorum,file.children.2.driver=raw"

Though very repetitive, it is also very simple: all children are almost
symmemtrical (identical in user data). The only thing for user/management tool
to make sure is the images have the same data.

Unfortunately the logic is more complicated in an persistent incremental backup
scenario. Manual users will have to specify bitmap file names and the
granularities which they may have no clue anymore two weeks after they created
the bitmap, and can get wrong.  Management seems a must in this case, but the
interface we provide to them still feels way too low level. Anyway, I do think
we can consider a "banana" (dummy name) driver for persistent bitmap management
which is structured like quorum:

    banana="driver=raw,file.driver=banana,file.mode=synchronous"
    banana="$banana,file.image.file.filename=$TEST_IMG"
    banana="$banana,file.bitmaps.0.file.filename=$TEST_DIR/bm0.raw"
    banana="$banana,file.bitmaps.0.granularity=65536"
    banana="$banana,file.bitmaps.0.name=bm0"
    banana="$banana,file.bitmaps.1.file.filename=$TEST_DIR/bm1.raw"
    banana="$banana,file.bitmaps.1.granularity=1048576"
    banana="$banana,file.bitmaps.1.name=bm1"
    ...

But we're merely inlining the information from QBM JSON format into the command
line. This is IMO only one step of differences in between. Compared to the
command line building blocks, QBM has:

Pros:
  - Better data integrity as all bitmaps are tied together by the descriptor
    file.
  - Ease of use because of qemu-img supportability.
  - Management or user can rely on the QBM file to store the bitmap information
    and query easily.
  - If any information need to be transactionally stored together with the
    bitmap data, it is easier to do as file writes, compared to QMP event and
    command communication between whoever is storing it. For example, think
    about implementing the in_use feature like qcow2 bitmaps.

Cons:
  - Maintenance burden of another format specification.
  - Redundancy with qcow2 internal bitmap format (unavoidable if we agree on
    raw).

Thus the QBM format makes sense to me.

Fam

[1]: http://docs.ceph.com/docs/hammer/rbd/qemu-rbd/
[2]: http://docs.ceph.com/docs/master/rbd/rbd-openstack/