Re: Update on tree-sitter structure navigation

[Danny Freeman]

Dmitry Gutov <dgutov@yandex.ru> writes:

> Hi Yuan,
>
> On 02/09/2023 08:01, Yuan Fu wrote:
>> - Solve the grammar versioning/breaking-change problem: tree-sitter grammar 
>> don’t have a version
>> number, so every time the author changes the grammar, our queries break, and 
>> loading the mode only
>> produces a giant error.
>
> I don't have a better idea than basically copying NeoVim and others: to 
> maintain the urls to parser
> repositories and the ref of the latest known good revision, for the current 
> version of the major
> mode. That info could be filled in by major modes themselves, e.g. in an 
> autoload block (similarly
> to how auto-mode-alist is appended to).

clojure-ts-mode keeps a URL for the parser, but doesn't do anything
about the git revision. It easily could but I don't feel the need (yet)
since I am also a maintainer of the clojure grammar and know when we're
about to break grammar consumers.

It's not quite that simple though. Some distributions (nixos for
example) are already providing pre-compiled grammars. That is how I
discovered a couple recent bugs in js-ts-mode, because the grammars
distributed with nixos 23.05 no longer worked on Emacs 30 after a patch
was applied that was supposed to be backwards compatible (a real pain to
verify in my experience).

With the way Emacs can load a grammar provided by the user's
distribution, keeping information about the version of the grammar in
the major mode doesn't help all that much. Even if we did it we have no
idea what version might be have been built used the user's
.emacs.d/tree-sitter folder. That would require something like putting a
version number in the file name, or maybe applying a patch to the
grammar's C source that allowed us to get a version, SHA, something at
runtime. 

I'm not so sure we can have a great way to do this without a change to
the tree-sitter libraries. I would love to see some kind of increasing
version number generated in the grammar's C source that we could then
access. It could be used to make decisions about what queries to use, or
to warn the user they need to use a different grammar (maybe offering to
install a compatible version).

Tree-sitter grammar changes are almost always breaking changes. Adding
nodes can break things, re-naming them and removing them definitely can.
I'm not sure any grammar consumer has a great way to deal with this
without always compiling the exact grammar they need and only ever using
it.

-- 
Danny Freeman