Re: fts in gnulib behave different than glibc

[Jim Meyering]

On Thu, Jul 29, 2021 at 3:22 AM Simon Josefsson <simon@josefsson.org> wrote:
>
> Jim Meyering <jim@meyering.net> writes:
>
> > On Wed, Jul 28, 2021 at 1:08 AM Simon Josefsson via Gnulib discussion
> > list <bug-gnulib@gnu.org> wrote:
> >> Hi.  I replaced GNU InetUtils' old custom fts implementation with the
> >> one from gnulib, but self-tests started failing.  Looking at the code,
> >> it seems gnulib's fts implementation has diverged compared to glibc, and
> >> has some optimizations that (I think) change the API (wrt stat and
> >> chdir).  Also, gnulib's fts module is always enabled, even on modern
> >> glibc systems.  InetUtils's usage of fts works fine with modern glibc,
> >> but it didn't work with gnulib's version (it needed a FTS_NOCHDIR).  The
> >> gnulib manual for the fts replacement module isn't terribly clear about
> >> this.  Is there a reason for this behaviour?
> >>
> >> I would prefer if there were two fts modules in gnulib:
> >>
> >> 1) One module 'fts' based on glibc's code, that is only enabled in on
> >> systems that doesn't have fts, or where fts is known to be buggy.
> >>
> >> 2) One 'fts-faster' that is the current code, which can be used when you
> >> always wants to pull in the optimized implementation.
> >>
> >> Then InetUtils would use system fts on glibc platforms, and not always
> >> have to pull in a replacement copy.
> >>
> >> What do you think?
> >>
> >> I could live with a new module 'fts-optional' that only pulls in the
> >> current 'fts' module when the system is lacking it.  That doesn't fix
> >> the API confusion, but is probably sufficient for InetUtils.
> >
> > There are fundamental flaws in the ABI of glibc's fts that make it
> > unsuitable for use in any tool I care about.
>
> Ouch -- is there disagreement from the glibc people on fixing glibc fts?
> Maybe the reaction will be different now.

It's intrinsic in the current ABI. There would have to be a new interface.
I chatted with Uli about this many years ago, and he would have been
happy to receive a patch adding the new interfaces, but gnulib's fts
solved all of my problems, so I never made time to add this to glibc.

> Are the problems inherent with the glibc ABI, or can they be fixed?  If
> it isn't possible to fix, maybe the entire API should be declared
> deprecated and eventually removed.
>
> The glibc manual doesn't seem to document fts?!
>
> > Those flaws make it easy to hit silly limits or to provoke inordinate
> > resource usage/DoS.
>
> It would be nice to document these problems in more detail in the gnulib
> manual.
>
> Is there any known behavioural difference between glibc fts and gnulib
> fts?  Documenting that too would be useful.

Yes, see each item on the list quoted below.
I think I can dig up some documentation on gnulib's fts.

> InetUtils' required a FTS_NOCHDIR flag in order to continue behave as
> before (a simple command like 'ls foo' where foo is a directory failed).
> I don't see any self-tests in gnulib without that flag, so maybe this
> suggests there is some API/ABI difference.
>
> > Is it ok for InetUtil's fts to be unable to do these things? (each of
> > which afflicts glibc fts, from what I recall)
> > - process files in a tree more than 2^16 levels deep
> > - detect certain cycles efficiently
> > - delete (in reasonable time) a hierarchy with too many entries in a
> > single directory.
>
> InetUtils only uses fts for "DIR" in ftpd, when it emulate /bin/ls
> internally (based on some old BSD implementation of /bin/ls that uses
> fts).  The first two applies, but not the third, I think, but it sounds
> like corner-cases.

They are corner cases, indeed.

Correcting that last item: s/delete/traverse/:
- traverse (in reasonable time) a hierarchy with too many entries in a
single directory.