Re: [Qemu-devel] QCOW2 support for LZO compression

[Peter Lieven]

Am 26.06.2017 um 11:33 schrieb Denis V. Lunev:
> On 06/26/2017 12:20 PM, Peter Lieven wrote:
> > Am 26.06.2017 um 10:28 schrieb Kevin Wolf:
> > > [ Cc: qemu-devel; don't post to qemu-block only! ]
> > >
> > > Am 26.06.2017 um 09:57 hat Peter Lieven geschrieben:
> > > > Hi,
> > > >
> > > > I am currently working on optimizing speed for compressed QCOW2
> > > > images. We use them for templates and would also like to use them for
> > > > backups, but the latter is almost infeasible because using gzip for
> > > > compression is horribly slow. I tried to experiment with different
> > > > options to deflate, but in the end I think its better to use a
> > > > different compression algorithm for cases where speed matters. As we
> > > > already have probing for it in configure and as it is widely used I
> > > > would like to use LZO for that purpose. I think it would be best to
> > > > have a flag to indicate that compressed blocks use LZO compression,
> > > > but I would need a little explaination which of the feature fields I
> > > > have to use to prevent an older (incompatible) Qemu opening LZO
> > > > compressed QCOW2 images.
> > > >
> > > > I also have already some numbers. I converted a fresh Debian 9 Install
> > > > which has an uncomressed QCOW2 size of 1158 MB with qemu-img to a
> > > > compressed QCOW2.  With GZIP compression the result is 356MB whereas
> > > > the LZO version is 452MB. However, the current GZIP variant uses 35
> > > > seconds for this operation where LZO only needs 4 seconds. I think is
> > > > is a good trade in especially when its optional so the user can
> > > > choose.
> > > >
> > > > What are your thoughts?
> > > We had a related RFC patch by Den earlier this year, which never
> > > received many comment and never got out of RFC:
> > >
> > > https://lists.gnu.org/archive/html/qemu-devel/2017-03/msg04682.html
> > I was not aware of that one. Thanks for pointing out.
> >
> > > So he chose a different algorithm (zstd). When I asked, he posted a
> > > comparison of algorithms (however a generic one and not measured in the
> > > context of qemu) that suggests that LZO would be slightly faster, but
> > > have a considerable worse compression ratio with the settings that were
> > > benchmarked.
> > My idea to choose LZO was that it is widely available and available in
> > any distro you can think of. We already have probing for it in configure.
> > My concern with ZSTD would be that it seems there are no packages
> > available for most distros and that it seems to be multi-threaded. I
> > don't
> > know if this will cause any trouble?
> We have had that compression working in multithreaded process.

Do you have a patchset ready ?

> > > I think it's clear that if there is any serious interest in compression,
> > > we'll want to support at least one more algorithm. What we still need to
> > > evaluate is which one(s) to take, and whether a simple incompatible flag
> > > in the header like in Den's patch is enough or whether we should add a
> > > whole new header field for the compression algorithm (like we already
> > > have for encryption).
> >  From my side there clearly is interest in optimizing the compression. Its
> > even possible to speed up zlib by 3-4x times by choosing other parameters
> > for deflate which unfortunately are not compatible with our inflate
> > settings.
> >
> > I don't know if its worth creating a new header field. Even if we
> > spent to bits
> > in the end (one for LZO and one for ZSDT). I think this wouldn't hurt.
> > However,
> > there are likely to pop up new compression algorithms in the future and
> > a header would be more flexible.
> >
> > I just don't want to make it too complicated and as you pointed out
> > compression is
> > not that interesting for most people - maybe due to its speed.
> I think we need something generic but simple. I think that we should not
> support compression with the different algorithm in the single file.

Agreed. The compression algorithm should be specified at create time.

> Speaking about compression, we do have different constraints for
> different situation, f.e. backups are written once and rarely read while
> generic compression in backing store is read frequently but never
> read. Thus the exact algorithm should be selectable.

Okay, so we have gzip (default) and zstd. If the patches for ZStd are ready
I don't mind to drop support for LZO. I think both ZStd and LZO both offer
good compression and decompression speed compared to gzip.

Peter