Re: RFC: fixing the 32-bit size and time limits in gzip file format

[Mark Adler]

On Aug 19, 2010, at 2:54 PM, Paul Eggert wrote:
> In that case I'm afraid that we need to give up on the goal of always
> providing a correct uncompressed length.  At this point the gzip
> format is so widely used that an incompatible change to it would cause
> far more trouble than the relatively minor problem of gzip -l
> reporting the wrong length.  Instead, it might be better to leave the
> format alone, and to change gzip -l so that it decompresses the data
> in order to report the uncompressed data length.

Paul,

That makes sense, except I keep hearing about this all the time.  For well over 
a decade.  Very often it also leads to the incorrect conclusion that gzip 
doesn't support > 4 GB files.  Always decompressing the whole thing would be 
rather slow, especially when it matters ( > 4 GB).

So I'm thinking we should put forth a format amendment.  However initially only 
decoding the format would be supported, not creating it.  We would let that 
simmer for, say, three years for the updated gzip, pigz, and zlib to free 
range.  Then let loose versions that create the format once the decoding has a 
wide distribution.

I used to think that eventually this would all go away since the gzip format 
would surely be supplanted by something else.  However even with .bz2 and .xz 
formats with better compression, that simply hasn't happened.  So now I'm 
beginning to think that the .gz format will stubbornly persist at least until I 
die (at which point I won't care anymore).

So how would a format amendment be put forth?  As an addition to the existing 
RFC?  As a new RFC?  Who knows how to do that?

By the way, I keep getting bounce messages from Peter's address, so I don't 
think he's getting these emails.

Mark