Re: [Qemu-devel] QCow2 compression

[Eric Blake]

On 02/26/2016 10:00 PM, address@hidden wrote:
> Hello, I am hoping someone here can help me. I am implementing QCow2 support 
> for a PC emulator project and have a couple questions regarding compression I 
> haven't been able to figure out on my own.

[Can you convince your mailer to wrap long lines? It makes replying to
your mail easier]

> 
> First some background:
> I am using the information I found at 
> https://people.gnome.org/~markmc/qcow-image-format.html

That page is outdated.  Better is the current spec, maintained directly
in qemu.git:

http://git.qemu.org/?p=qemu.git;a=blob;f=docs/specs/qcow2.txt

> and I have implemented working support for QCow2 images as described there 
> except for snapshots, encryption, and compression. Of these, only compression 
> is of immediate use to me.

Do NOT implement encryption, at least not in the form documented in
anything you've read so far.  We have a pending patch to implement LUKS
encryption, to replace the insecure existing encryption, although Dan's
latest email says it might not land until qemu 2.7:

https://lists.gnu.org/archive/html/qemu-devel/2016-02/msg06552.html


> 
> I have some QCow2 images all using 16-bit clusters created using qemu-img 
> 2.1.2

You may want to use current qemu.git, which will soon be 2.5, although
compression hasn't really changed since then.

> According to the documentation I linked,

Better, according to the current documentation:

> Compressed Clusters Descriptor (x = 62 - (cluster_bits - 8)):
>
>   Bit 0 - x: Host cluster offset. This is usually _not_ aligned to a
>              cluster boundary!
>
>    x+1 - 61: Compressed size of the images in sectors of 512 bytes

> an L2 entry value of 4A C0 00 00 00 3D 97 50.

So with default 64k clusters, x = 62 - (16 - 8) = 54.  Bits 0-54 are the
host cluster offset, or 0x003d9750, but that is in terms of host
sectors.  The comment in block/qcow2.c is telling, and perhaps we should
improve the qcow2 spec to make it obvious:

  - Size of compressed clusters is stored in sectors to reduce bit usage
    in the cluster offsets.

Thus, in your image, the guest compressed data starts at sector
0x003d9750, or host file offset 0x7b2ea000.  This value is NOT aligned
to a cluster, but IS aligned to a sector (since a sector is the smallest
unit we write to), and makes more sense than something ending in 0x50
(which is not sector aligned).

> This would lead me to believe the cluster starts at offset 0x3D9750
and has a length of 0x2B 512-byte sectors (or 0x2B times 0x200 = 0x5600).

You are correct about the 64k of guest data being compressed into 0x5600
bytes in the host file, but incorrect at where to read those bytes.

> 
> A final question: I noticed that compressed clusters typically have a 
> reference count higher than one, yet there are no snapshots present in the 
> image. I suspect the count is incremented for each compressed cluster that 
> exists even partially within a normal sized cluster region of the file, but I 
> can find no documentation to this effect and am merely speculating. Am I 
> correct?

Yes, you are correct, and yes, the spec I pointed to documents that.
Since the L2 entry in your example occupies only 43/128 sectors, any
other adjacent clusters from the guest point of view can be compressed
into the remaining sectors of the host cluster, and the refcount must be
equal to the number of all compressed guest clusters that occupy at
least one sector of the host cluster.

> 
> If it is the wrong place to ask these questions, I would appreciate it if 
> anyone could direct me to a more appropriate venue. Thank you for taking the 
> time to read this and tanks in advance for any assistance.

This is the right list.


-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org

signature.asc
Description: OpenPGP digital signature