Re: gzip --rsyncable via pigz?

[Mark Adler]

Paul,

On Jun 12, 2013, at 12:31 PM, Paul Eggert <address@hidden> wrote:
> pigz has many improvements over gzip.
> In particular, its --rsyncable implementation appears
> to be better than the Debian / Fedora implementation
> of gzip --rsyncable.

I wrote pigz's rsyncability to be better, but this is the first confirmation 
I've heard that I succeeded.  How do you know it's better?

> A downside is that pigz does not generate output that is
> byte-for-byte the same as gzip output.

I intend for pigz to replace gzip.  The only downside is that the first time 
you use pigz instead of gzip for your rsynced stuff, it will take longer for 
that one rsync.  After that it will work as intended, and you now have the 
advantage of the greater speed of pigz.

> Another downside is that, in my tests, pigz -9 generates
> output that's a bit longer than gzip's.

That is because pigz breaks the input into blocks that can be recombined on 
byte boundaries, adding a few bytes for each block.  The effect is small.  pigz 
is designed to produce exactly the same output regardless of whether multiple 
processors are used or not.

For larger block sizes, pigz can produce smaller output than gzip (the default 
block size is 128K):

% gzip -9 < linux-3.1.6.tar | wc -c
 97360066

% pigz -9 < linux-3.1.6.tar | wc -c
 97368984

% pigz -9 -b 256 < linux-3.1.6.tar | wc -c
 97352064

% pigz -9 -b 1024 < linux-3.1.6.tar | wc -c
 97339330

Also if you don't care how long it takes, you can use -11:

% pigz -11 < linux-3.1.6.tar | wc -c
 92892986

Mark