Re: RFC: fixing the 32-bit size and time limits in gzip file format

[Greg Roelofs]

Mark wrote:

> Eventually new versions of gzip that don't issue that warning in
> that case would replace the old ones.  We could stick in a small
> extra field that simply indicates that those extra four bytes
> are there.  (We'd have to always accept that hit in the streaming
> case, since you wouldn't know until too late that you needed
> the longer length.  The total hit would be eight bytes -- four
> for the extra field and four for the extended length at the end.)

The extra field is a minimum of six bytes (assuming there are no other
subfields already present).

> Concatenated gzip streams with older gzips unaware of the extra
> field would still be supported by preventing the two-byte magic
> header (1f 8b) in the appended length extension.  Simply shift
> the 32-bit value up one bit, forcing the low bit to be zero.
> Then it can't match 8b1f on the bottom.  That limits the total
> length to 63 bits, but that's not really a loss since the off_t
> type is signed anyway.

Concatenated gzip streams made by newer gzips fail on older gzips,
though.  I'm not sure I believe that's a worthwhile tradeoff.

Greg