Re: [Qemu-block] QCOW2 support for LZO compression

[Denis V. Lunev]

On 06/26/2017 11:28 AM, Kevin Wolf wrote:
> [ Cc: qemu-devel; don't post to qemu-block only! ]
>
> Am 26.06.2017 um 09:57 hat Peter Lieven geschrieben:
>> Hi,
>>
>> I am currently working on optimizing speed for compressed QCOW2
>> images. We use them for templates and would also like to use them for
>> backups, but the latter is almost infeasible because using gzip for
>> compression is horribly slow. I tried to experiment with different
>> options to deflate, but in the end I think its better to use a
>> different compression algorithm for cases where speed matters. As we
>> already have probing for it in configure and as it is widely used I
>> would like to use LZO for that purpose. I think it would be best to
>> have a flag to indicate that compressed blocks use LZO compression,
>> but I would need a little explaination which of the feature fields I
>> have to use to prevent an older (incompatible) Qemu opening LZO
>> compressed QCOW2 images.
>>
>> I also have already some numbers. I converted a fresh Debian 9 Install
>> which has an uncomressed QCOW2 size of 1158 MB with qemu-img to a
>> compressed QCOW2.  With GZIP compression the result is 356MB whereas
>> the LZO version is 452MB. However, the current GZIP variant uses 35
>> seconds for this operation where LZO only needs 4 seconds. I think is
>> is a good trade in especially when its optional so the user can
>> choose.
>>
>> What are your thoughts?
> We had a related RFC patch by Den earlier this year, which never
> received many comment and never got out of RFC:
>
> https://lists.gnu.org/archive/html/qemu-devel/2017-03/msg04682.html
>
> So he chose a different algorithm (zstd). When I asked, he posted a
> comparison of algorithms (however a generic one and not measured in the
> context of qemu) that suggests that LZO would be slightly faster, but
> have a considerable worse compression ratio with the settings that were
> benchmarked.
>
> I think it's clear that if there is any serious interest in compression,
> we'll want to support at least one more algorithm. What we still need to
> evaluate is which one(s) to take, and whether a simple incompatible flag
> in the header like in Den's patch is enough or whether we should add a
> whole new header field for the compression algorithm (like we already
> have for encryption).
>
> Kevin
I have been contacted today Yann Collet who is ZSTD maintainer, he has
dropped
nowadays status of ZSTD, which could be useful for the discussion:

"_1. zstd package availability_

We have been tracking distribution availability since Zstandard official
release, in September 2016 :
https://github.com/facebook/zstd/issues/320
There is also this tool which tracks availability of packages :
https://repology.org/metapackage/zstd/versions

zstd seems now available as a package in most recent distributions.
It’s even part of “core” for recent BSD releases.
Zstandard v1.0 is still less than 1 year old, so older distributions
typically do not have it (or support a development version).
That’s the main limitation currently. We expect things to improve over time.

 2.

    _Compression speed is good but does not matter
    _For such scenarios, it’s possible to trade speed for more compression.
    At its maximum compression level (--ultra -22), zstd compression
    ratio (and speed) is close to lzma.
    A nice property though is that decompression speed remains roughly
    the same at all compression levels,
    about 10x faster than lzma decompression speed (about 1 GB/s on
    modern CPU).

 3.

    _zstd is multi-threaded, and it’s dangerous_

libzstd is single-threaded.
There is a multi-thread extension, which is enabled in the CLI, but not
in the library.
There is also an experimental target which makes it possible to produce
a MT-enabled library.
Even in this case, the API remains single-threaded by default.
It’s necessary to use dedicated entry points to enable multi-threading.
TL;DR : zstd supports multithreading, but is single threaded by default.

 

 4.

    _How to identify gz format from zstd one ?
    _Many implementations assume they require to add some custom header
    in order to identify gz from zstd.
    That’s not the case: well-formed compression format already provide
    a header with enough information to guarantee their identity.
    Such “good” compression format include gz, zstd, xz, lz4-frame, to
    name a few.
    For zstd, the identifier is a 4-bytes value, documented in the
    compression format :
    
https://github.com/facebook/zstd/blob/dev/doc/zstd_compression_format.md#zstandard-frames
    As an example, zstd project provides a zlib-wrapper which is able to
    dynamically recognize an input as gz or zstd, and route to
    appropriate decoder, without any special header :
    https://github.com/facebook/zstd/tree/dev/zlibWrapper


Unfortunately, not all compression algorithm do provide unambiguous
standard header.
LZO, for example, does not by default.
Behind a single name, lzo effectively groups multiple incompatible
variants, which must be correctly identified for proper decoding."

Den