Re: [Lzip-bug] lzip memory & performance issue

[markk]

Hi,

Antonio Diaz Diaz wrote:
> address@hidden wrote:
>> lzip's initial memory check seems to be too conservative. Or does lzip
>> uses more memory than xz and lzma_alone with the same dictionary size?
>
> IIRC lzip 1.11 does use a few MB more than lzma-utils for the same
> dictionary size. I don't know the amount used by other lzma_alone
> implementations.
>
> Lzip 1.11 does not perform any memory check. It tries to assign the
> needed memory, and if it can't, it prints the "Not enough memory. Try a
> smaller dictionary size." message. I think xz does perform a memory
> check and adjusts the dictionary size accordingly.

xz doesn't perform a check by default. (You can specify
--memlimit-compress= to tell it to do that.)


>  From http://www.nongnu.org/lzip/manual/lzip_manual.html#Introduction
> "The amount of memory required for compression is about 5 MiB plus 1 or
> 2 times the dictionary size limit (1 if input file size is less than
> dictionary size limit, else 2) plus 8 times the dictionary size really
> used. The option `-0' is special and only requires about 1.5 MiB at most."

For a 256MiB dictionary then, lzip should have used about 2565MiB
(compared to 2405MiB for a 240MiB dictionary). However, the lzip manual
says:
  "Note that dictionary sizes are quantized. If the specified size does
not match one of the valid sizes, it will be rounded upwards."

What are the relevant sizes that the dictionary is quantized to? Could
lzip have been trying to use a much larger dictionary size when I
specified 256MiB?


>> So for that specific file, lzip took about 2.5 times as long as xz.
>
> I guess your system is swapping slightly because of the large dictionary
> size. As you can see, for example here[1], lzip and xz are comparable
> both in compression time and size, with "lzip -9" tending to be slower
> than "xz -9" but compressing a little more.
> [1] http://martin-steigerwald.de/computer/programme/packbench/index.html
>
>
>> Any idea what causes the time difference? Could
>> different compiler options have that large an effect? I think I compiled
>> xz with -O3 but lzip with -O2.
>
> I have only noticed small speed changes in lzip between -O2 and -O3. I
> don't know the effects on xz, but I don't think they can be so large.

The difference wasn't due to swapping. Can anyone else reproduce the
difference I'm seeing?

I tested lzip again, this time with a smaller, more compressible file (a
tar archive, 633169920 bytes long). I used a 128MiB dictionary size to
rule out the possibility of swapping being the cause of lzip being slower
than xz.

Again, there was a significant difference between the compression speed of
lzip and xz, with lzip taking about 2.5 times as long as xz. The
xz-compressed file was slightly smaller than the lzip one.

$ time lzip --verbose --keep -9 --dictionary-size=128MiB file.tar
  file.tar:  1.891:1,  4.230 bits/byte, 47.12% saved, 633169920 in,
334811557 out.

real    19m8.546s
user    19m4.836s
sys     0m3.296s

$ time xz -v --compress --keep -9e --lzma2=dict=128MiB file.tar
file.tar (1/1)
  100 %       318.5 MiB / 603.8 MiB = 0.527   1.3 MiB/s       7:47

real    7m47.825s
user    7m44.165s
sys     0m2.760s

$ du --bytes file.tar.xz
333996208       file.tar.xz