Re: [Lzip-bug] Re: performance: gzip, lzip, xz

[Jim Meyering]

Antonio Diaz Diaz wrote:
> Maybe xz has a clear goal, but I have been unable to discover what it
> could be. Perhaps its goal is to find out the limit between format
> flexibility and format security, given the number of times the xz
> format had to be changed due to security problems.

The only changes after the first official .xz format specification have
been language fixes (typos etc.), not technical changes.  If in doubt,
look at section "0.3. Version History" or use diff to compare the
versions.  Development versions varied a lot, of course, that's the
nature of development.  Development versions did have some security
issues, but those were fixed before the first stable release of the
specification, made on 2009-01-14.

> Clearly long term stability is not the goal of xz. Just read the
> README file for 4.999.9beta, line 51:
> "Since the .xz format allows adding new filter IDs, it is possible
> that some day there will be a filter that is, for example, much faster
> to compress than LZMA2 (but probably with worse compression
> ratio). Similarly, it is possible that some day there is a filter that
> will compress better than LZMA2".

The words you quote do not support your assertion.

> Will the old filters be removed as new ones are added, leaving users
> without support for old files, or will xz become increasingly bloated
> by old filters that almost nobody uses?

Perhaps you're imagining that filters will be added willy-nilly,
without a thought to long-term usability and the possibility of bloat.
With "if false, ..." you can conclude anything.

...
>> The .xz format is in no way an archive-like format. You cannot store
>> file names in .xz, and .xz supports even less metadata than .gz.
>
> By archiver-like I mean it is way too complicated for a general
> purpose compressor and it includes features I have only found in
> archiver formats, like the subblock filter.

Can you name archiver formats that specify subblock-like filters?

...
>> Claiming long-term stability of the .lz format is a stretch.
...
>> The file format has changed at least once (probably twice, but I'm
>> not sure) since the first stable release.  Older versions of lzip
>> cannot decompress new format files.  The same can and (I'm sure) will
>> happen with .xz too, but in case of .lz, it has been about adding basic
>> features that .xz had in the first place.
>
> Lzip format has changed exactly once form the first released
> version. The only two changes were:
> The "member size" field was added to improve the recovery of undamaged
> members from multimember files.
> Coding of dictionary size in member header was extended to support
> more fine grained values.

lzip 1.5 supports the version 1 file format, but it does not support the
sync flush marker, which lzip 1.7 and later support.  Look for "Sync Flush
marker" in decoder.cc.  Equivalent code is not present in 1.5.