bug#51982: Erroneous handling of local variables in byte-compiled nested lambdas

[Stefan Monnier]

> (Really want to get rid of `let*`!)

FWIW, we could get rid of it in `cconv-convert`.
That would have less impact than doing it in macroexpansion.

[ We could also force dynamically-scoped code to go through (a neutered
  version of) cconv.el , so that bytecomp.el and byte-opt.el can presume
  that `let*` doesn't exist any more.  ]

Tho maybe we'd want to eliminate `let` instead.  ;-)

BTW, have you checked the impact on byte-code quality?

>> These two tests are identical aren't they?
> No, they exercise different code paths (let and let*).

Then that deserves a comment ;-)

>> Looks good (better than patch A).
>
> And here I was prepared to apply patch A since it's slightly more
> conservative and it seems to be a rare problem anyway.
> I've now split the patches in a more sensible (and easily reviewed) way: the
> first corresponds to patch A, and the second is the diff to B. Take a second
> look before making up your mind.
>
>> You say "On the other hand, patch B does abuse the cconv data structures
>> a little (but it works!)" so the code should say something about
>> this abuse.  A least I failed to see where the abuse lies.
>
> There are comments and doc strings such as
>
>   EXTEND is a list of variables which might need to be accessed even
>   from places where they are shadowed, because some part of ENV causes
>   them to be used at places where they originally did not
>   directly appear.
>
> but with the B patch we put things into `extend` that are not strictly
> variables but (international-get-closed-var N).

See below, I think we don't need to put them there.

> Similarly, `env` has entries like (VAR . (apply-partially F ARG1 ARG2 ..))
> where the ARGi are always treated as variables but now they can be access
> forms as well.

I don't think the current code assumes that ARGs are vars here.
You're probably right that it used to be the case and it's not any more,
but that shouldn't cause problems.  The risk I can see is if one of
those ARGs is an expression which refers to a var which gets shadowed,
in which case `cconv--remap-llv` won't rewrite it the way it should.
But I think with your code ARG will either be a simple var or something
of the form (internal-get-closed-var N) so we should be safe.

> @@ -304,6 +304,22 @@ cconv--convert-funcbody
>              `(,@(nreverse special-forms) ,@(macroexp-unprogn body))))
>        funcbody)))
>  
> +(defun cconv--lifted-arg (var env)
> +  "The argument to use for VAR in λ-lifted calls according to ENV."
> +  (let ((mapping (cdr (assq var env))))
> +    (pcase-exhaustive mapping
> +      (`(internal-get-closed-var . ,_)
> +       ;; The variable is captured.
> +       mapping)
> +      (`(car-safe (internal-get-closed-var . ,_))
> +       ;; The variable is mutably captured; skip
> +       ;; the indirection step because the variable is
> +       ;; passed "by reference" to the λ-lifted function.
> +       (cadr mapping))
> +      ((or '() `(car-safe ,(pred symbolp)))
> +       ;; The variable is not captured; use the (shadowed) variable value.
> +       var))))

The docstring or comment at the beginning should mention this function
is specifically for shadowed vars.

Also, If mapping is of the form (car-safe SYMBOL) is `var` really the
correct answer?  Shouldn't it still be (cadr mapping)?

>  (defun cconv-convert (form env extend)
>    ;; This function actually rewrites the tree.
>    "Return FORM with all its lambdas changed so they are closed.
> @@ -428,10 +444,11 @@ cconv-convert
>                   ;; One of the lambda-lifted vars is shadowed, so add
>                   ;; a reference to the outside binding and arrange to use
>                   ;; that reference.
> -                 (let ((closedsym (make-symbol (format "closed-%s" var))))
> +                 (let ((var-def (cconv--lifted-arg var env))
> +                       (closedsym (make-symbol (format "closed-%s" var))))
>                     (setq new-env (cconv--remap-llv new-env var closedsym))
>                     (setq new-extend (cons closedsym (remq var new-extend)))
> -                   (push `(,closedsym ,var) binders-new)))
> +                   (push `(,closedsym ,var-def) binders-new)))

Looks good.
Side note: I don't understand why we `(cons closedsym`, since that
`closedsym` can never appear in another binding (since it's fresh).

[ Version B: ]

> @@ -441,14 +442,16 @@ cconv-convert
>                         (cconv-convert value env extend)))))
>  
>                 (when (and (eq letsym 'let*) (memq var new-extend))
> -                 ;; One of the lambda-lifted vars is shadowed, so add
> -                 ;; a reference to the outside binding and arrange to use
> -                 ;; that reference.
> -                 (let ((var-def (cconv--lifted-arg var env))
> -                       (closedsym (make-symbol (format "closed-%s" var))))
> -                   (setq new-env (cconv--remap-llv new-env var closedsym))
> -                   (setq new-extend (cons closedsym (remq var new-extend)))
> -                   (push `(,closedsym ,var-def) binders-new)))
> +                 ;; One of the lambda-lifted vars is shadowed; if
> +                 ;; necessary, add a reference to the outside binding
> +                 ;; and arrange to use that reference.
> +                 (let* ((lifted-arg (cconv--lifted-arg var env)))
> +                   ;; This means that we may add accessors to ENV and EXTEND
> +                   ;; passing them off as variables, but it's close enough.
> +                   (setq new-env (cconv--remap-llv new-env var lifted-arg))
> +                   (setq new-extend (cons lifted-arg (remq var new-extend)))
> +                   (when (symbolp lifted-arg)
> +                     (push `(,lifted-arg ,var) binders-new))))

Just like I think the `(cons closedsym` is useless in the current (and
in patch A), I think this `(cons lifted-arg` is not needed.

I don't much like this `symbolp` test (which fundamentally seems to
be trying to recover the information about which branch of the `pcase`
we're coming from in `cconv--lifted-arg`).  It at least deserves
a comment explaining why it's doing the right thing.
If we can remove this `symbolp` test recovering info about provenance of
the result of `cconv--lifted-arg` then I think option B is better, but
I prefer otherwise option A.


        Stefan