[Lzip-bug] response to decoder error by lzip-1.2-pre3

[John Reiser]

Hi Antonio,

When decompressing, it seems that lzip-1.2-pre3 does not write
the partial output buffer upon encountering an error in the compressed
input.  Usually this discards some correct data that has been decoded
before the error was detected.  It can be difficult to determine the
exact length of the tail of decoded data which is incorrect,
but nevertheless not writing any of the partial output buffer
almost always discards some correct data.  Even the incorrect tail
can be useful.  I believe that the partial output buffer should
be written.  Both the shell exit status of "failure", and the message
on stderr, are enough indication of non-success.

Example:
  cp COPYING foo   # our favorite uncompressed file
  ./lzip foo   # compress it
  cat foo.lz foo.lz >bar.lz   # two copies, one after the other
  <<Use a binary file editor to clobber 4 bytes of bar.lz to zero,
    at offset 0x2000, which is within the first copy of foo.lz.
    (gdb) x/4x 0x2000
    0x2000:     0x98a44d50      0x34d2ff94      0x252f6398      0xe75db6a3
    (gdb) set *(int *)0x2000=0
    (gdb) x/4x 0x2000
    0x2000:     0x00000000      0x34d2ff94      0x252f6398      0xe75db6a3
  >>
  ./lzip -dc bar.lz >gorp   # attempt to decompress the damaged concatenation
    bar.lz: decoder error at pos 9214   # error message from lzip
  ls -l gorp
  -rw-rw-r-- 1 jreiser jreiser 0 2008-12-08 10:29 gorp   # empty file

Also notice that there was no error recovery.  The second half of the
compressed data file bar.lz contains an intact and correct compressed
stream of foo.lz, but this was not found and was not decoded.  I would
like to use a decoder which is more robust.  The decoder should be able
to recover from an error within one item of a concatenation,
and correctly decode all non-damaged members.

In particular, consider the Z_FULL_FLUSH parameter of zlib.h.  This
is exactly what I want.  At compress time: align compressed data to
next byte boundary (fill with zero bits as necessary), write all pending
bytes so that a decoder which is next in a shell pipeline can decode
everything so far, insert a marker into the compressed stream that
can be recognized as a recovery point, and reset compression state
(erase history, clear dictionary) so that a decoder can start
decompressing successfully after random access to this point in the
compressed stream.

Regards,

-- 
John Reiser