lzip-bug
[Top][All Lists]
Advanced

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Lzip-bug] On Windows unpacking does not use all cores


From: Antonio Diaz Diaz
Subject: Re: [Lzip-bug] On Windows unpacking does not use all cores
Date: Thu, 01 Mar 2018 23:03:17 +0100
User-agent: Mozilla/5.0 (X11; U; Linux i586; en-US; rv:1.9.1.19) Gecko/20110420 SeaMonkey/2.0.14

Romano wrote:
May I ask one last thing, is there a reason to use LC - literal
context parameter lower than 8? Default is 3 and in lzma2 its not
even allowed above 4. But lc8 give better ratio(~2.2% in my tests and
many others on fileforums) for most data especially game repacks. I
changed parameter in the header and it seems workign fine(I guess
your lzlib is based on lzma1?), but I never understood why would 3 be
the default promoted choice especially since lc does not affect speed
and seems to be only benefit at 8.

The reason to not use lc = 8 is that it requires 768 KiB of fast RAM vs the 24 KiB required by lc = 3. This would make decompression slow on machines with, say, 64 KiB of cache. It also does not increase the compression ratio of tarballs by much. I just made a little test with a bunch of tarballs:

Total size of tar files = 2_649_872_384
Compressed sizes:
level         -0             -6             -9
lc = 3    576_393_065    391_449_171    367_964_614
lc = 8    569_160_271    388_354_250    365_461_202
             -1.2%          -0.8%          -0.7


Lastly, pls consider to add more lzma options to the command line in
next versions. Specifically mc and lc parameter would be very useful.

Adding backward-incompatible options is out of the scope of lzip. As you can read here[1], "The lzip format is designed for long-term archiving. Therefore it excludes any unneeded features that may interfere with the future extraction of the decompressed data."

[1] http://www.nongnu.org/lzip/manual/lzip_manual.html#Quality-assurance

There are lots of tweaks that can increase slightly the compression ratio of some kinds of data, generally at the cost of increasing resource consumption (memory/cpu), and worse, at the cost of producing incompatible files.


There is also one tweak that 4x4 use that have significant effect on
speed of compression: it test incoming data that are to be processed by
lzma with some fast "order 0 model" first. If it cannot compress better
than to 99% then data are copied instead. This give anywhere between
50%-300% speed increase.

These clever tweaks are fine for experimenting, but are also out of the scope of a stable standard general-purpose compressor like lzip. See for example how neither gzip nor bzip2 have made such changes to their formats.


Best regards,
Antonio.



reply via email to

[Prev in Thread] Current Thread [Next in Thread]