Re: Anything better for delayed lexical evaluation than (lambda () ...)?

[David Kastrup]

Mark H Weaver <address@hidden> writes:

> Replying to myself:
>
>> Andy Wingo <address@hidden> writes:
>>> It is my opinion -- and I could be wrong here, either by
>>> misunderstanding (again!) the issues, or for whatever reason -- that
>>> closures are the best solution to the #{#} problem.
>>
>> I would love to use closures for this, but they simply are not powerful
>> enough to implement `local-eval' properly.
>>
>> First of all, closures cannot capture the syntactic environment, so
>> things like (let-syntax ((foo ...)) (the-environment)) would not work.
>
> This statement (and the rest of my previous email), may have been based
> on a mistaken impression of your "closure" proposal.  I assumed that you
> meant that (the-environment) should essentially implemented as a macro
> that expands into a (lambda () ...) that would handle every possible
> lexical variable access or mutation.
>
> However, based on the IRC logs, now I think you may have meant that
> Lilypond should _scan_ the Lily code between #{ and #} looking for
> embedded Scheme code, _before_ evaluating anything.
>
> If this could be done reliably, it would certainly make our lives
> easier.  Lilypond could create the entire Scheme expression with all the
> embedded bits of Scheme as closures (not general purpose ones, but with
> the exact right code within).  We could compile the resulting code, and
> life would be good.

To be fair: this is what we currently do, and we only actually call
those lambdas that end up actually being recognized by the grammar.  So
as long as (primitive-eval `(lambda () ,(read))) is guaranteed to not
ever choke, the potential for error is limited.  And to be fair: we
already need to use the Scheme reader for skipping stuff behind # and $
or it would be hard to reliably detect the matching #} for the #{
construct when it occurs nested within Scheme again.

It took a number of iterations to make all this robust and dependable,
and just recently a Lilypond comment of the form
% (this can be either #UP or #DOWN)
caused trouble since the Scheme reader not just read #DOWN but also )
and then unread it again, making it end up at the start of the _next_
embedded expression for which the Scheme reader was asked for its
opinion.

> I don't know enough about Lilypond's language to know whether it is
> feasible to do this robustly.

So-so, but then we don't get around _scanning_ (and thus reading) the
potential sexps in a dry run anyway with the current design.  This would
be different if Lilypond employed a different delimiter strategy.

> We have already established that Lily cannot be parsed without runtime
> information.  Furthermore, their language needs to use lexical
> tie-ins, which means that the lexer needs contextual information from
> the parser.  Therefore, they cannot even produce a stream of lexical
> tokens prior to execution.
>
> So, with no help from either the parser or scanner, it sounds like
> they'd be stuck with something like the usual Emacs strategy of
> scanning source code with regexps to do syntax highlighting: it works
> in practice most of the time, but it will fail sometimes for unusual
> inputs.  That's fine for a syntax highlighter, but a language
> implementation should be more robust.

Because of the delimiting issue, there is actually not much of a gain to
be expected.  Results would improve in cases like

#{ % A Lilypond comment before a Scheme element #(
   #y %) and the comment ends again
#}

In this case, the current behavior will be that #y has no (lambda ()...)
associated with it, and consequently evaluation will be done without a
lexical environment, and fail.  The old version using local-eval would
have worked, but it would still have required the sexp starting at #( to
end somewhere before #}.

> In summary, it makes my head hurt to even think about how I would
> implement and maintain Lilypond on Guile 2.0 using this strategy.

I posted a link to the working code already.  It made my head hurt.

> It shouldn't be this difficult.  This case with Lilypond has convinced
> me that `the-environment' and `local-eval' can be very useful, and
> sometimes they cannot be easily replaced with our other facilities.

I think that it is a good thought experiment to assume Lilypond _could_
be properly compiled.  To get lexical closure, we would have to bounce
_everything_ around in uncompiled or macro form until the top-level form
got assembled.  For debugging and error messages, that would be
inconvenient.  Again, to be fair: the current code has to associate file
names and line numbers with the string in order to get useful error
messages.  So it took work getting errors to throw where they belong.

So at its current state, Lilypond itself does not much more than
lukewarmly ask for local-eval.  But for the sake of GUILE, I would
consider it a mistake to lose sight of it.  It is a powerful tool in the
language embedding and self-awareness department.

-- 
David Kastrup