[Lzip-bug] lzip memory & performance issue

[markk]

Hi,

I've been experimenting with lzip, xz and lzma_alone, and have come across
a couple of issues with lzip 1.11.

lzip's initial memory check seems to be too conservative. Or does lzip
uses more memory than xz and lzma_alone with the same dictionary size?

My system: x86, 4GB RAM, running Xubuntu 9.10 32-bit with PAE kernel. With
both xz and lzma_alone, I can use a 256MiB dictionary, and still have
enough memory to e.g. use Firefox and other applications without swapping.
(Using the non-PAE kernel I can still use that dictionary size, but
running any other applications in addition to the Xfce desktop causes a
lot of swapping.)

However, if I run lzip specifying a dictionary size of 256MiB, it aborts
after a few seconds, reporting
  Not enough memory. Try a smaller dictionary size.
Reducing the dictionary size to 240MiB does allow lzip to run.


The compression speed of lzip 1.11 seems significantly slower than that of
xz 5.0.1. On one large, not very compressible file:
$ time lzip --verbose --keep -9 --dictionary-size=240MiB bigfile.tar
  bigfile.tar:  1.200:1,  6.668 bits/byte, 16.65% saved, 2586347520 in,
2155794974 out.

real    111m33.927s
user    110m48.688s
sys     0m22.145s

xz took 44:07 to compress the same file doing
$ xz -v --compress --keep -9e --lzma2=dict=256MiB bigfile.tar

So for that specific file, lzip took about 2.5 times as long as xz. The
xz-compressed file turned out slightly smaller than the lzip one;
2154653168 bytes. Any idea what causes the time difference? Could
different compiler options have that large an effect? I think I compiled
xz with -O3 but lzip with -O2.