Re: [Zutils-bug] zstd support?

[Sven Hartrumpf]

Antonio wrote 2018-10-06 01:41:
> Daniel Baumann wrote:
>> On 10/05/2018 06:55 PM, Antonio Diaz Diaz wrote:
>>> Has zstd improved since the last time I tried it[1][2]?
>>
>> In the context of zutils, I don't think it matters if zstd is good or
>> not. zstd is available and people are using it, thereore zutils imho
>> should cover it.
>>
>> (otherwise, with that logic zutils should not support xz either, right?)
> 
> Xz was not supported by the first version of zutils. Support was added
> later because other GNU projects were (misled to) using it. Even so,
> support for xz is partially optional. Both ztest and zupdate ignore xz
> files if no xz decompressor is installed. I even have the hope that I'll be
> able to remove xz support from zutils someday.
> 
> In addition to having a format that makes xz look good in comparison, zstd
> seems an order of magnitude more difficult to support than any of the four
> formats already supported. It does not provide an unique magic string, and
> supporting dictionary decompression may be difficult. Zstd is all about
> performance at any price for the needs of Facebook. Just like brotli is
> tuned to the needs of Google.
> 
> I would prefer to not mislead the users of zutils by supporting a format
> like zstd that considers the checksum an informative field that can be
> ignored. "We [software developers] have a moral duty to choose wisely the
> formats we use because everybody else will blindly use whatever formats we
> choose"[1].

I agree that the use cases for zstd are special;
if zutils is intended to be only archive-oriented, you should choose
wisely - as you do.

> [1] http://www.nongnu.org/lzip/xz_inadequate.html#why
> 
> In any case, I'm now so delayed with all my other projects that I can't add
> support for any format to zutils in the next several months.

I see.
I found a different solution for a zgrep alternative that deals with
special compression formats:
rg (ripgrep, see https://github.com/BurntSushi/ripgrep )
plus its --pre option, which calls a helper for each file to be grepped.
A suitable helper script looks like this:

  case "$f" in
    *.zst)
       zstd -cdqq $f
       ;;
    *.gz)
       gzip -cd $f
       ;;
    *.bz2)
       bzcat $f
       ;;
...
    *)
       cat $f
       ;;
  esac


Ciao
Sven