bug#30626: 26.0.91; Crash when traversing a `stream-of-directory-files'

[John Mastro]

Noam Postavsky <npostavs@gmail.com> wrote:
> Aha, but the following also crashes, whether compiled or not:
>
> ;; -*- lexical-binding: t -*-
>
> (require 'stream)
>
> (let* ((stream0 (stream-range 1 1000000))
>        (stream stream0))
>   (while (not (stream-empty-p stream))
>     (cl-callf stream-rest stream)))
>
> So the problem is that the initial stream0 object can reach the entire
> unfolding stream as it goes, and just holding on to that reference is
> enough to keep the whole thing in memory.
>
> Now, I can see that letting stream0 automagically get access to the
> unfolded result can be an optimization in some cases, although in this
> case it's a pessimization.  It could also affect the semantics if
> unfolding the stream has side effects, not sure if stream.el makes
> guarantees about that though.

Clojure has/had a similar issue. They describe this scenario (having a
live reference to the beginning of the stream, preventing GC from
collecting it) as "holding onto the head" of the stream (in Clojure
they're called lazy seqs).

Their solution was what they call "locals clearing". The compiler tracks
the lifetimes of local bindings and "clears" them (by setting them to
nil/null) after their point of last use, e.g.:

(let* ((stream0 (stream-range 1 1000000))
       (stream stream0))
  (setq stream0 nil) ;; <<< Inserted by compiler
  (while (not (stream-empty-p stream))
    (cl-callf stream-rest stream)))

If the code does reference stream0 later, locals clearing can't help
you, but that's considered a "if it hurts, don't do it" situation.

This probably isn't practical for Emacs, especially since it could only
work for byte-compiled code, but thought the prior art may be of
interest.

        John