Re: [Lzip-bug] lzip memory & performance issue

[Antonio Diaz Diaz]

Hello Mark,

address@hidden wrote:
> lzip's initial memory check seems to be too conservative. Or does lzip
> uses more memory than xz and lzma_alone with the same dictionary size?

IIRC lzip 1.11 does use a few MB more than lzma-utils for the same 
dictionary size. I don't know the amount used by other lzma_alone 
implementations.

Lzip 1.11 does not perform any memory check. It tries to assign the 
needed memory, and if it can't, it prints the "Not enough memory. Try a 
smaller dictionary size." message. I think xz does perform a memory 
check and adjusts the dictionary size accordingly.

From http://www.nongnu.org/lzip/manual/lzip_manual.html#Introduction
"The amount of memory required for compression is about 5 MiB plus 1 or 
2 times the dictionary size limit (1 if input file size is less than 
dictionary size limit, else 2) plus 8 times the dictionary size really 
used. The option `-0' is special and only requires about 1.5 MiB at most."


> So for that specific file, lzip took about 2.5 times as long as xz.

I guess your system is swapping slightly because of the large dictionary 
size. As you can see, for example here[1], lzip and xz are comparable 
both in compression time and size, with "lzip -9" tending to be slower 
than "xz -9" but compressing a little more.
[1] http://martin-steigerwald.de/computer/programme/packbench/index.html


> Any idea what causes the time difference? Could
> different compiler options have that large an effect? I think I compiled
> xz with -O3 but lzip with -O2.

I have only noticed small speed changes in lzip between -O2 and -O3. I 
don't know the effects on xz, but I don't think they can be so large.


Regards,
Antonio.