Re: tree-sitter: conceptional problem solvable at Emacs' level?

[Lynn Winebarger]

On Sat, Feb 11, 2023 at 4:34 AM Ihor Radchenko <yantar92@posteo.net> wrote:
>
> Eli Zaretskii <eliz@gnu.org> writes:
>
> > Another possibility is to complicate the function we pass to
> > tree-sitter with which to read buffer text, in a way that replaces the
> > text of a macro with something else (in the simplest case, just space
> > characters), so as to avoid errors in the parser, and again analyze
> > the macros in our own code.
>
> Another idea is delegating parts of buffer to Elisp/alternative parser.
>
> Tree sitter provides support to documents written using a mixture of
> grammars: 
> https://tree-sitter.github.io/tree-sitter/using-parsers#multi-language-documents
> Macros can be considered such a "mixed" grammar with macros being a
> grammar of their own.
>
> AFAIU, tree sitter allows excluding certain file ranges from parsing
> and instead parse the excluded ranges using alternative grammar. If
> Elisp can somehow tell tree-sitter backend not skip parsing
> macro-looking lines, it should solve the problem at least partially.
>
What's needed is (a) a generic "macro keyword" terminal in tree-sitter
grammar, recognized by the lexer because they can appear anywhere, and
(b) a parser for macro definitions.
Then (b) maintains the set of macros that the lexer uses to recognize
instances of (a).
For extra credit, the macros could be hypothetically expanded, the
results parsed, and the annotations generated on instances of the
macro arguments mapped back to their occurrence as arguments.  Maybe
some kind of "unfold" notation could be used to see the results of the
expansion and the resulting annotations in context.

Lynn