Re: Export checkpoint/bitmap from image on qcow2

[Nir Soffer]

On Fri, Dec 22, 2023 at 3:35 PM João Jandre Paraquetti
<joao@scclouds.com.br> wrote:
>
> Hello Nir,
>
> Thank you for your detailed response, but this process does seem a bit
> too complex.
>
> I've been thinking, wouldn't it be easier to use the process below?
>   - Create a paused transient dummy VM;
>   - Create the necessary checkpoints on the VM using dumps from the
> original VM;

Not sure what do you mean by dumps from the original vm...

>   - Use Libvirt's virDomainBackupBegin to create the incremental backup;
>   - Destroy the dummy VM.
>
> What do you think?

It should work and the backup code will be simpler, since you use the
libvirt APIs for both
live and "cold" backup.

But creating the vm in paused mode and integrating this new mode into
your system can
be more complicated than the cold backup code, depending on your system.

Some issues you will have to think about:
- when a vm is running in the backup mode, does it consume resources
(e.g. memory) that it does not need?
- how do you switch to normal mode if a user want to start a vm in the
middle of a backup? (full backup of a large vm can take hours)
- can you migrate a vm during backup?

In oVirt we went in the other way - we never use live backup -
instead, we do this
for every backup:

1. create a snapshot
2. backup the vm using the snapshot (instead of the active image)
3. delete the snapshot

With this both live and cold backup are the same, and we avoid the
issue when long backup
(e.g full backup, or incremental backup with a lot of data) blocks
usage of the vm during the
backup. You can start/stop/migrate a vm while it is being backed up
and the backup can run
on another host in the cluster (if using shared storage).

Nir

> On 12/18/23 18:44, Nir Soffer wrote:
> > On Thu, Nov 30, 2023 at 4:14 PM João Jandre Paraquetti
> > <joao@scclouds.com.br> wrote:
> >> Hi, all
> >>
> >> I recently started looking into how to create incremental backups using
> >> Libvirt+Qemu. I have already found that, since Libvirt 7.6.0, we can use
> >> the virDomainBackupBegin API to create incremental backups of live VMs
> >> through Libvirt. Using this API we can back up the VM's disks and create
> >> a checkpoint at the same time. After the first full backup, we can
> >> create incremental backups referencing the checkpoint that was generated
> >> on the previous backup process.
> >>
> >> As far as I understood Libvirt's documentation, when using this API to
> >> create a checkpoint, what happens is that Libvirt will create a new
> >> bitmap on the VM's disk to track the changes that happened from that
> >> checkpoint on, then, they'll copy what changed between the last bitmap
> >> and the new one. By doing this, Libvirt is able to create incremental
> >> backups of disks that are used by running VMs.
> >>
> >> My problem is the following though: after I stop the VM, I'm no longer
> >> able to use Libvirts API as it requires the VM (domain) to be running.
> >> However, there might still be changes on the disk since the last backup
> >> I performed. Therefore, to create a backup of the changes made since the
> >> last backup until the VM was stopped, I have to work directly on the VM
> >> image using Qemu. I have checked, and the QCOW2 file has the checkpoint
> >> created with the Libvirt backup process (I checked via the "qemu-img
> >> info"). Therefore, we have the marker of the last checkpoint generated
> >> by the last backup process. However, if the VM is stopped, there is no
> >> clear way to export this last differencial copy of the volume.
> >>
> >> I have searched through the documentation about this; however, I haven't
> >> found any example or explanation on how to manually export a
> >> differencial copy of the QCOW2 file using the last checkpoint (bitmap)
> >> created.
> >>
> >> I would much appreaciate, if people could point me to the direction on
> >> how to export differencial copies of a QCOW2 file, using the checkpoints
> >> (bitmaps) that it has.
> > Backup is complicated and a lot of hard work. Before you reinvent the
> > wheel, check
> > if this project project works for you:
> > https://github.com/abbbi/virtnbdbackup
> >
> > If you need to create your own solution of want to understand how backup 
> > works
> > with bitmaps, here is how it works.
> >
> > The basic idea is - you start an nbd server exporting the dirty
> > bitmap, so the client can
> > get the dirty extents using NBD_BLOCK_STATUS and copy the dirty clusters.
> >
> > To start an nbd server you have 2 options:
> >
> > - qemu-nbd using the --bitmap option. vdsm code is a good place to
> > learn how to do this
> >    (more about this below). This was the best option few years ago, but
> > now you may want
> >    the second option.
> >
> > - qemu-storage-daemon - you configure and start the nbd server in the same
> >    way libvirt does it - using the qmp commands. libvirt code is
> > probably the best place
> >    to learn how to do this. This is probably the best option at this
> > time, unless you are
> >    using an old qemu version that does not have working qemu-storage-daemon.
> >
> > If you have a single qcow2 image with the bitmap, you are ready to
> > export the image.
> > But you may have a more interesting image when the same bitmap exists
> > in multiple
> > snapshots. You can see all the bitmaps and snapshots using:
> >
> >      qemu-img info --backing-chain --output json filename
> >
> > If the bitmap exists on multiple images in the chain, you need to
> > validate that the bitmap
> > is valid - it must exist in all images in the chain since the bitmap
> > was created and must have
> > the right flags (see vdsm code below for the details).
> >
> > After validating the bitmap, you need to create a new bitmap with all
> > the bits in the same bitmap
> > in all the images in the chain. You have 2 ways to do this:
> >
> > - with qemu-nbd: create an overlay on top of the image, create a
> > bitmap in the overlay, and merge
> >    the bitmaps from the image chain into this bitmap. Then start
> > qemu-nbd with the overlay. You can
> >    create and merge bitmaps using `qemu-img bitmap ...` (see vdsm code
> > below for the details)
> >
> > - with qemu-storage-daemon: you can do what libvirt does - create a
> > new temporary non-persistent bitmap
> >    (in qemu-storage-daemon memory), and merge the other bitmaps in the
> > chain using qmp commands.
> >    (see libvirt source or debug logs for the details).
> >
> > When the backup bitmap is ready, you can start the nbd server and
> > configure it to export this bitmap.
> >
> > When backup is done delete the overlay or the temporary backup bitmap.
> >
> > You can check how vdsm does this using qemu-nbd here:
> > https://github.com/oVirt/vdsm/blob/0fc22ff0b81d605f10a2bc67309e119b7462b506/lib/vdsm/storage/nbd.py#L97
> >
> > The steps:
> >
> > 1. Finding the images with the bitmap and validating the bitmap
> >     
> > https://github.com/oVirt/vdsm/blob/0fc22ff0b81d605f10a2bc67309e119b7462b506/lib/vdsm/storage/nbd.py#L217
> >
> > 2. Creating overlay with a backup bitmap
> >     
> > https://github.com/oVirt/vdsm/blob/0fc22ff0b81d605f10a2bc67309e119b7462b506/lib/vdsm/storage/nbd.py#L184
> >
> > 3. Starting qemu-nbd with exporting the bitmap using systemd-run:
> >     
> > https://github.com/oVirt/vdsm/blob/0fc22ff0b81d605f10a2bc67309e119b7462b506/lib/vdsm/storage/nbd.py#L419
> >
> > Nir
> >
>