Re: [Lzip-bug] plzip vs. lzip: different decompression size

Hi Antonio,

I had some time to play around with this a bit longer and could reproduce without dd or my backup image. You might find the below interesting.

The two plzip decompressions ended (were killed) prematurely due to "Out of memory" issues.

I've attached the kern.log extracts which were automatically generated.

# dd if=/dev/zero of=zerofile.txt count=10240 bs=1048576

# plzip -k -v -o devZeroFile.lz devZeroFile.txt

devZeroFile.txt: 6800.655:1, 0.001 bits/byte, 99.99% saved, 10737418240 in, 1578880 out.

# plzip -vvdc devZeroFile.txt.lz > devZeroFile_plzip_take1.txt

devZeroFile.txt.lz: Killed

# plzip -vvdc devZeroFile.txt.lz > devZeroFile_plzip_take2.txt

devZeroFile.txt.lz: Killed

# lzip -vvdc devZeroFile.txt.lz > devZeroFile_lzip.txt

devZeroFile.txt.lz: version 1, dictionary size 8 MiB. done

<snip>

# md5sum devZeroFile*

2dd26c4d4799ebd29fa31e48d49e8e53 devZeroFile_lzip.txt

00f0a382428fc047e73c1b17f469af40 devZeroFile_plzip_take1.txt

1a672b7b4fd545569f6bd83442eabd5a devZeroFile_plzip_take2.txt

2dd26c4d4799ebd29fa31e48d49e8e53 devZeroFile.txt

b8a5254603f1140d8a1d38fb30cba92d devZeroFile.txt.lz

# ls -al

total 18803952

drwxr-xr-x 3 root root 4096 Jan 15 19:02 .

drwxr-xr-x 23 root root 4096 Jan 5 11:14 ..

-rw-r--r-- 1 root root 10737418240 Jan 15 19:45 devZeroFile_lzip.txt

-rw-r--r-- 1 root root 4216856576 Jan 15 18:48 devZeroFile_plzip_take1.txt

-rw-r--r-- 1 root root 4299317248 Jan 15 18:53 devZeroFile_plzip_take2.txt

-rw-r--r-- 1 root root 10737418240 Jan 15 18:28 devZeroFile.txt

-rw-r--r-- 1 root root 1578880 Jan 15 18:28 devZeroFile.txt.lz

-rw-r--r-- 1 root root 23716 Jan 15 18:51 kern.log__take1

-rw-r--r-- 1 root root 11789 Jan 15 18:55 kern.log__take2

drwx------ 2 root root 16384 Jan 5 11:15 lost+found

For your information, I've also tried with pbzip2, and there the issue does not occur.

# pbzip2 -k -v devZeroFile.txt

Parallel BZIP2 v1.1.8 - by: Jeff Gilchrist [http://compression.ca]

[Jun. 10, 2012] (uses libbzip2 by Julian Seward)

Major contributions: Yavor Nikolov <address@hidden>

# CPUs: 2

BWT Block Size: 900 KB

File Block Size: 900 KB

Maximum Memory: 100 MB

-------------------------------------------

File #: 1 of 1

Input Name: devZeroFile.txt

Output Name: devZeroFile.txt.bz2

Input Size: 10737418240 bytes

Compressing data...

Output Size: 572687 bytes

-------------------------------------------

Wall Clock: 212.419426 seconds

# pbzip2 -vvdc devZeroFile.txt.bz2 > devZeroFile_pbzip2_take1.txt

Parallel BZIP2 v1.1.8 - by: Jeff Gilchrist [http://compression.ca]

[Jun. 10, 2012] (uses libbzip2 by Julian Seward)

Major contributions: Yavor Nikolov <address@hidden>

# CPUs: 2

Maximum Memory: 100 MB

Ignore Trailing Garbage: off

-------------------------------------------

File #: 1 of 1

Input Name: devZeroFile.txt.bz2

Output Name: <stdout>

BWT Block Size: 900k

Input Size: 572687 bytes

Decompressing data (no threads)...

-------------------------------------------

Wall Clock: 175.498747 seconds

# pbzip2 -vvdc devZeroFile.txt.bz2 > devZeroFile_pbzip2_take2.txt

Parallel BZIP2 v1.1.8 - by: Jeff Gilchrist [http://compression.ca]

[Jun. 10, 2012] (uses libbzip2 by Julian Seward)

Major contributions: Yavor Nikolov <address@hidden>

# CPUs: 2

Maximum Memory: 100 MB

Ignore Trailing Garbage: off

-------------------------------------------

File #: 1 of 1

Input Name: devZeroFile.txt.bz2

Output Name: <stdout>

BWT Block Size: 900k

Input Size: 572687 bytes

Decompressing data (no threads)...

-------------------------------------------

Wall Clock: 143.202060 seconds

# md5sum devZeroFile_pbzip2_take*.txt

2dd26c4d4799ebd29fa31e48d49e8e53 devZeroFile_pbzip2_take1.txt

2dd26c4d4799ebd29fa31e48d49e8e53 devZeroFile_pbzip2_take2.txt

Regards,

Filip

On Tue, Jan 7, 2014 at 8:51 PM, Antonio Diaz Diaz <address@hidden> wrote:

Hello Filip.

Filip Van Lerberge wrote:

I've performed a few tests and it appears that plzip and lzip have
different behavior when it comes down to decompression:

Feedback messages are different because lzip is usually used to decompress single-member files while plzip is more often used to decompress multi-member files. Just give a single -v to lzip when decompressing multi-member files.

But of course the decompressed output produced by lzip and plzip should be identical, and I constantly check very carefully that this is the case:

$ plzip -cvv /dev/fd0 > img.lz
/dev/fd0: 8.917:1, 0.897 bits/byte, 88.79% saved, 1474560 in, 165360 out.
$ lzip -cdvv img.lz | md5sum
img.lz: 8.917:1, 0.897 bits/byte, 88.79% saved. done
ecb5f45b5c02b11b44f961e6afbce7c3 -
$ plzip -cdvv img.lz | md5sum
img.lz: 8.917:1, 0.897 bits/byte, 88.79% saved. done
ecb5f45b5c02b11b44f961e6afbce7c3 -

# *plzip *-vvdc Archimedes_vda_2014-01-02_18-26-27.img.lz | dd
of=/lv/archi-plzip.img

Have you tried without dd?:
plzip -vvdc Archimedes_vda_2014-01-02_18-26-27.img.lz > /lv/archi-plzip.img

-rw-r--r-- 1 root root 5492916224 Jan 5 11:18 archi-plzip.img
-rw-r--r-- 1 root root 107374182400 Jan 5 11:54 archi-lzip.img

You may try to compare the two files to see if the first is a truncated version of the second. I suspect there is a broken pipe somewhere and the plzip output gets truncated as a result.

The version of lzip in Ubuntu 13.10 is 1.14 (vs. 0.9 for plzip). Is this
the root cause of the issue I'm experiencing?

I don't think so. AFAIK, no version of plzip has had any "real" bug (one producing incorrect output).

Any plans to update plzip and to update the Ubuntu package?

I don't know about Ubuntu, but Debian already provides the latest version of plzip (1.1) in experimental:

http://packages.debian.org/experimental/plzip

Best regards,
Antonio.

From:	Filip Van Lerberge
Subject:	Re: [Lzip-bug] plzip vs. lzip: different decompression size
Date:	Wed, 15 Jan 2014 19:56:09 +0100